Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning
Authors: Théo Vincent
Fabian Wahren
Jan Peters
Boris Belousov
Carlo D'Eramo
Venue: ICLR 2025 Poster (2025)
Keywords: automated reinforcement learning
deep reinforcement learning
hyperparameter selection
Primary area: reinforcement learning
TL;DR: A novel approach for automated deep reinforcement learning that selects adaptively the target of the RL agent at each update to tackle the non-stationarity of the optimization procedure.

Abstract: Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called _Adaptive $Q$-Network_ (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several $Q$-functions, each one trained with different hyperparameters, which are updated online using the $Q$-function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari $2600$ games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability. Our code is available at *https://github.com/theovincent/AdaDQN*.

Published as a conference paper at ICLR 2025
ADAPTIVE Q-NETWORK: ON-THE-FLY TARGET SELECTION FOR DEEP REINFORCEMENT LEARNING
Théo Vincent 1.2,*
Fabian Wahren 1.2 Jan Peters! 1.2,3
Boris Belousov!
Carlo D'Eramo2:3,4
DFKI GmbH, SAIROL
Department of Computer Science, TU Darmstadt *Center for Al and Data Science University of Würzburg
3 Hessian.ai, TU Darmstadt
ABSTRACT
Deep Reinforcement Learning (RL) is well known for being highly sensitive hyperparameters, requiring practitioners substantial efforts to optimize them for the problem hand. This also limits the applicability of RL in real-world scenar- ios. In recent years, the field of automated Reinforcement Learning (AutoRI.) has grown popularity by trying address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-effcciency and practicality. Furthermore, most AutoRL meth- ods are heavily based on already existing AutoML methods, which were orig- inally developed neglecting the additional challenges inherent RL due to its non-stationarities. In this work, we propose new approach for AutoRL, called Adaptive Q-Network (AdaQN), that is tailored to RL to take into account the non- stationarity of the optimization procedure without requiring additional samples. AdaQN learns several Q-functions, each one trained with different hyperparam- eters, which are updated online using the Q-function with the smallest approx- imation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate in MuJoCo control problems and Atari 2600 games. showing benefits in sample- efficiency, overall performance, robustness to stochasticity and training stability. Our code is available at https: //git thub. com/ wtt.tthevvvvneeeeee ncent /AdaDoN.
INTRODUCTION
Deep Reinforcement Learning (RL) has proven effective at solving complex sequential decision problems in various domains (Mnih et al., 2015; Haarnoja et al., 2018: Silver et al., 2017). De- spite their success in many fields, deep RL algorithms suffer from brittle behavior with respect to hyperparameters (Mahmood al., 2018; 8; Henderson et al., 2018). For this reason, the field of au- tomated Reinforcement Learning (AutoRL) (Parker-Holdes et al., 2022) has gained popularity in recent years. AutoRL methods aim to optimize the hyperparameter selection so that practitioners can avoid time-consuming hyperparameter searches. AutoRL methods also seek to minimize the number of required samples to achieve reasonable performances so that RL algorithms can be used in applications where limited number of interactions with the environment is possible. AutoRL is still its early stage of development, and most existing methods adapt techniques that have been shown to be effective for automated Machine Learning (AutoML) (Hutter et al., 2019; Falkner et al., 2018). However, RL brings additional challenges that have been overlooked till now, despite notori- ously requiring special care (Igl et al., 2021). For example, due to the highly non-stationary nature of RL, it is unlikely that a static selection of hyperparameters works optimally for a given problem and algorithm (Mohan et al., 2023). This has led to the development of several techniques for adapting the training process online to prevent issues like local minima (Nikishin et al., 2022; Sokar et al., 2023), lack of exploration (Klink et al., 2020), and catastrophic forgetting (Kirkpatrick al., 2017). In this work, we focus on the line of work in the AutoRL landscape that aims at dynamically chang- ing the hyperparameters during training achieve good performances from scratch as opposed to approaches that aim at finding promising static hyperparameters to use in a sub-sequent training.
*Correspondance theo. incentdffki

---

Published as a conference paper at ICLR 2025
Q- Q-Network
Adaptive Q-Network (ours)
rQ 0 TQQ TQQ
rQ
-QQ - Q/TQ
12-0Q/QQ--0
Figure I: Left: Each line represents a training of Q-Newwork (QN) with different hyperparameters. Right: At the jt target update, Adaptive Q-Network (AdaQN) selects the network Q, (highlighted with crown) that is the closest to the previous target I where I' is the Bellman operator.
We introduce novel approach for AutoRL to improve the effectiveness of learning algorithms by coping with the non-stationaritiis of the RL optimization procedure. Our investigation stems from the intuition that the effectiveness of each hyperparameter selection changes dynamically after each training update. Building upon this observation, we propose to adaptively select the best- rest-performing hyperparameters configuration to carry out Bellman updates. To highlight this idea, we call our ap- proach Adaptive Q-Nelwork (AdaQN). In practice, AdaQN uses an ensemble of Q-functions, trained with different hyperparameters, and selects the one with the smallest approximation error (Schaul et al., 2015; D' Eramo & Chalvatzaki, 2022) to be used as shared target for each Bellman update. By doing this, our method considers several sets of hyperparameters at once each time the station- arity of the optimization problem breaks, i.e., at each target update (Figure 1). In the following, we provide a theoretical motivation for our selection mechanism and then, we empirically validate AdaQN in MuJoCo control problems and Atari 2600 games, showing that it has several advantages over carrying out individual runs with static hyperparameters. Importantly, by trading off different hyperparameters configurations dynamically, AdaQN not only can match the performance of the best hyperparameter selection at no cost of additional samples, but can also reach superior overall performance than any individual run. Finally, we compare AdaQN against strong AutoRL baselines and evince AdaQN's superiority in finding good hyperparameters in sample-efficient manner.
2 PRELIMINARIES
We consider discounted Markov decision processes (MDPs) defined as = A, 7), where S and A are measurable state and action spaces, P S A + A(S) is a transition kernel, R. S x AA- + R is a reward function, and 7 [0, 1) is a discount factor (Puterman, 1990). A policy is a function π S + A(A), inducing an action-value function Q* a) - En 12 2'R(s,, 0s)|s0 = 8, ag = al that gives the expected discounted cumulative return execut- ing action in state following policy thereafter. The objective is to find an optimal policy T = arg max, (*(.), where (*(.)= = Eo-a( (ee=ar)\Q"(( a)]. Approximate value iteration (AVT) and approximate policy iteration (API) are two paradigms tackle this problem (Sutton & Barto, 1998). While AVI aims at estimating the optimal action-value function Q', i.e., the action-value function of the optimal policy, API is two-step procedure that alternates between Approximate policy evaluation (APE), a method to evaluate the action-value function Q" of the current policy m and policy improvement, which updates the current policy by taking greedy actions on Q".
Both AVI and APE aim to solve a Bellman equation, whose solution is Q" for AVI and Q for APE. Those solutions are the fixed point of a Bellman operator I", where I' is the optimal Bellman operator I' for AVI, and the Bellman operator D" for APE. For Q-function Q, state s and an action o, T' a) = r+>yErr-(ss.)) [max. Qss .²)] Q= a) rr (((())(((s')))s) a') d²)]- I' and -x are 7-contraction mapping in the infinite norm. Because of this, these two methods re- peatedly apply their respective Bellman operator, starting from a random Q-function. The fixed point theorem ensures that each iteration is closer to the fixed point of the respective Bellman equation. Hence, the more Bellman iterations are performed, the closer the Q-function will be to the desired one. In model-free RL, I' and Ix are approximated by their empirical version that we note f with- out distinguishing between AVI and APE since the nature of I can be understood from the context. We denote O the space of Q-functions parameters. In common Q-Network (QN) approaches, given a fixed vector of parameters 0 E θ, another vector of parameters 0 is learned to minimize
Lov(0 s, a, s') = (f a) Qo(s,a))²
AA (A())) denotes the set of prohahility measures over a set X.
2

---

Published as a conference paper at ICLR 2025
for sample (s, a, r, s'). Q3 is usually called the target Q-function because it is used to compute the target IQ2 while Qa is called the online Q-function. We refer to this step the projection step since the Bellman iteration I'Qq is projected back into the space of Q-functions that can be represented by our function approximations. The target parameters 0 are regularly updated to the online parameters I so that the following Bellman iterations are learned. This step is called the target update. We note 8 the target parameters after the 2th target update. In this work, at each target update i, we aim at finding the best hyperparameters Ç: characterizing the learnable parameters 0, such that
C = arg max J(O.) s.t. 0g E arg f(0: Ç. 0,-1).
(2)
where J is the cumulative discounted return and is the objective function of the learning process.
3 RELATED WORK
Methods for AVI APE (Mnih al., 2015; Dabney et al., 2018; Haarnoja et al., 2018; Bhatt al., 2024) are highly dependent on hyperparameters (Henderson et al., 2018; Andrychowicz et al., 2020; Engstrom et al., 2019). AutoRI tackles this limitation by searching for effective hyperparameters.
We follow the categorization of AutoRI methods presented in Parker-Holder et al. (2022) to posi- tion our work in the AutoRI landscape. Contrary to AdaQN, many approaches consider optimizing the hyperparameters through multiple trials. A classic approach is to cover the search space with a grid search or random search (Hutter et al., 2019; Bergstra & Bengio, 2012). Some other methods use Bayesian optimization to guide the search in the space of hyperparameters (Chen et al., 2018; Falkner et al., 2018; Nguyen et al., 2020; Shala et al., 2022), leveraging the information collected from individual trials. Those methods tackle the problem of finding the best set of hyperparameters but do not consider changing the hyperparameters during a single trial, which would be more ap- propriate for handling the non-stationarittes inherent to the RL problem. To this end, evolutionary approaches have been developed (Stanley et al., 2009; Jaderberg et al., 2019; Awad et al., 2021), where the best elements of a population of agents undergo genetic modifications during training. As an example, in SEARL (Franke et al., 2021), a population of agents is first evaluated in the environ- ment. Then, the best- pess-performing agents selected, and a mutation process is performed to form the next generation of agents. Finally, the samples collected during evaluation are taken from a shared replay buffer to train each individual agent, and the loop repeats. Even if selecting agents based on their performance in the environment seems reasonable, we argue that it hinders sample-efficiency since exploration in the space of hyperparameters might lead to poorly performing agents. More- over, if poorly performing agents are evaluated, low-quality samples will then be stored in the replay buffer and used later for training. which could lead to long-term negative effects. This is why we propose to base the selection mechanism on the approximation error, which does not require addi- tional interaction with the environment. Other methods have considered evaluating the Q-functions offline (Tang& & Wiens, 2021). Most approaches consider the Bellman error IITQ; Qi || instead of the approximation error |[ Qill, where Q: is the Q-function obtained after Bellman itera- tions (Farahmand & Szepesvári, 2011; I; Zitovsky et al., 2023). However, this approach considers the empirical estimate of the Bellman error which is a biased estimate of the true Bellman error (Baird, 1995). Lee al. (2022) propose to rely the approximation error choose between two different classes of functions in an algorithm called ModBE. AdaQN differs from ModBE since ModBE does not adapt the hyperparameters at each Bellman iteration but instead performs several training steps before comparing two classes of functions. Furthermore, the use of the approximation error can be theoretically justified by leveraging Theorem 3.4 from Farahmand (201 1), as shown in Section 4.
Meta-gradient reinforcement learning (MGRL) also optimizes the hyperparameters during train- ing (Finn et al., 2017: Zahavy al., 2020; Flennerhag et al., 2021). However. while MGRL fo- cuses on tuning the target to ease the optimization process, AdaQN uses diverse online networks to overcome optimization hurdles. This shift in paradigm leads AdaQN to avoid some of MGRL's shortcomings: AdaQN can work with discrete hyperparameters (e.g., optimizer, activation function, losses, or neural architecture), only requires first-order approximation, and does not suffer from non- stationarities when hyperparameters are updated since each online network is trained with respect to its own hyperparameters (Xu et al., 2018). Finally, AdaQN fits best in the "Blackbox Online Tun- ing" cluster even if most methods in this cluster focus on the behavioral policy (Schaul et al., 2019; Badia et al., 2020) or build the hyperparameters as functions of the state of the environment (Sutton & Singh, 1994; White& & White, 2016). For example, Riquelme et al. (2019) develop a method called
3

---

Published as a conference paper at ICLR 2025
adaptive TD that selects between the TD-update the Monte-Caut update depending whether the TD-update belongs to a confidence interval computed from several Monte-Carle estimates.
4 ADAPTIVE TEMPORAL-DIFFEREE
TARGET SELECTION
Notoriously, when the AVI/APE projection step described Section 2 is carried out for non-linear function approximation together with off-policy sampling, convergence to the fixed point is no longer guaranteed. Nevertheless, well-established theoretical results in AVI relate the approxima- tion error [[ Q || at each target update to the performance loss | QTT ||. i.e., the distance between the Q-function at timestep N during the training and the optimal one Q* For fixed number Bellman iterations N, Theorem 3.4 from Farahmand (2011), stated in Appendix A, demonstrates that the performance loss is upper bounded by term that decreases when the sum of approximation errors E ||L Qa, decreases, i.c., every projection step improves. Hence, minimizing the of approximation errors is crucial to get closer the fixed point. However, due to the complexity of the landscape of the empirical loss Lon (Mohan et al., 2023), using fixed fixe hy- perparameters minimize each approximation error [ Q is unlikely to be effective and can, in the worst casc, lead to local minima (Nikishin et al., 2022; Sokar et al., 2023) or diverging behaviors (Baird, 1995).
In this work, tackle this problem by introducing method that can seamlessly handle multiple hyperparameters in a single AVI/APE training procedure to speed up the reduction of the perfor- mance loss described above, compared call Adaptive Q-Network (AdaQN), learns several online networks trained with different hyperpa- rameters. Crucially, to cope with the non-stationarity of the RL procedure, each target update, the online network with the lowest approximation eтror and thus the lowest performance loss, is selected as a shared target network used train all the online networks. This results in moving all the networks of the ensemble towards the one that is currently the closest to the fixed point, thus benefiting from the heterogeneous ensemble speed up learning.
fixed hyperparameter selection. This method, which
We denote ( the parameters of the online networks and 0; the parameters of the shared target network after the 7th target update. To minimize the qh approximation error, the optimal choice for 0 given 0,-1, is the index of the closest online network from the target IQ
k==
0 H where và arg min Ke kE()...,
[TQū.-, Q 12,.:
where y is the distribution of state-action pairs in the replay buffer. Note that Equation (3) contains the true Bellman operator I' which is not known. Instead, we propose to rely on the value of the empirical loss select the next target network:
0: <- 8t where dp = arg min
Σ LoN (0x3,000-1
(4)
kE(I.....K) (s.a.T."")ED
where 'D is the replay buffer. Theorem 4.1 shows under which condition the selected index in Equation (4) is the same as the in Equation (3). Importantly, this condition is valid when the dataset is infinite and the samples are generated from the true distribution. Indeed, the empirical Bellman operator unbiased w.r.t. the true Bellman operator. The proof, inspired by the bias- variance trade-off in supervised learning, can be found in Appendix A.
Theorem 4.1. Let (6t) ) E OK and E be vectors of parameters representing K + 1 Q- of samples. Let v be the distribution represented by the
functions. Let D ((,,a,r,')) be state-action pairs present in DD We note De.a = ((r, 8') s')) r, r') E D), (), E D.
If for every state-action pair (s, a) E D, E(r,'')DD.. [Qals, a) = IQQass, a), then we have
arg arg min [[Q8 - ke kE
17=--Qo=1132 10 arg min ke(1..... (S.D.n.')
Σ
(,,,0,0,,,,,,,))),
Key to approach is that each vector of parameters 6 can be trained with different optimizer, learning rate, architecture, activation function, or any other hyperparameter that only affects its train- ing. By cleverly selecting the next target network from set of diverse online networks, AdaQN has
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Adaptive Deep Q-Network (AdaDQN), Modifications to DQN are marked in purple.
1: Initialize K online parameters (0* )== and an empty replay buffer Set = and 0 <- 0 the target parameters. Set the cumulative losses L = 0, for l: 1,..., K.
2: repeat
3: Set 16 U(L,..... K} w.p. ED and v6 s w.p. 1- Ed.
4: Take action 04 ~ (-greedy(Qua) (Quas ()), Observe reward rt, next state $1+1-
5: Update + DU(((, at, Tt; 8t+1)].
6: every G steps
7: Sample mini-batch B = {(s,[[ssssre r, from D. r, max
8:
Compute the shared target y
9:
for k:
= 1,..., K do
10:
Compute the loss w.r.t 8h, SN = (Gaa,n (y Qa+
Update La < LA + LUN-
12:
Update 8 using its specific optimizer and learning rate from ga LŌN-
13:
every T steps
14:
Update vp +- arg min, Lx; La <- 0, for k E (1,...,K}
15:
Update the target network with 0 <- 8".
higher chance of overcoming the typical challenges of optimization presented earlier. Importantly, we prove that AdaQN is guaranteed converge to the optimal aation-value function with probability 1 in a tabular setting in Appendix A.I.
4.1 ALGORITHIIC IMPLEMENTATION
Multiple algorithms be derived from formulation. Algorithm shows adaptive version of Deep Q-Network (DQN, Mnih et al. (2015)) that we call Adaptive Deep Q-Network (AdaDQN). Similarly, Adaptive Soft Actor-Critic (AdaSAC), presented in Algorithm 2, is an adaptive version of Soft Actor-Critic (SAC, Haarnoja et al. (2018)). In SAC, the target network is updated using Polyak averaging (Lillicrap al., 2015). Therefore, consider K target networks (0* ) )k=1- Each target network 8 is updated from its respective online network 8k, as shown in Line 11 of Algorithm 2. However, each online network is trained w.r.t. shared target chosen from a single target that comes from the of K target networks. Similarly to the strategy presented in Equation (4), after the :i target update, the shared target network is chosen as
0, t 8t where v0- arg min kE(I.....K) (s.a.r.a")ED
Σ
(5)
Multiple online networks enable to choose which network to use to sample actions in a value-based setting (see Line 3 of Algorithm 1) and which network is selected to train the actor in actor- critic setting (see Line 13 of Algorithm 2). This choice is related to the behavioral policy, thus, we choose the same strategy in both settings. Intuitively, the optimal choice would be to pick the best- performing network; however, this information is not available. More importantly, only exploring using the berf-pergo network could lead the other online networks learn passively (Sec- tion 5.1). This would make the performances of the other networks drop (Ostrovski et al., 2021), making them useless for the rest of the training. Inspired by c-greedy policies commonly used in RL for exploration, we select random network with probability еb and select the online network corresponding to the selected target network (as a proxy for the best-performing network) with prob- ability ég. We use linear decaying schedule for ég. Finally, at each target update, the running loss accumulated after each gradient step is used instead of the loss computed over the entire dataset.
5 EXPERIMENTS
We first evaluate our method when hyperparameters are selected from finite space. This setting is relevant when the RL practitioner has good understanding most hyperparameters and only hes- itates among few ones. We compare AdaQN to exhaustive grid search where both approaches have access to the same sets of byperparameters. This allows us to see how AdaQN performs com- pared to the best static set of hyperparameters. Additionally, the environment steps used for tuning the hyperparameters are also accounted for when reporting the results. This is why, for reporting the grid search results, multiply the number of interactions
focus sample efficiency; thus,
5

---

Published as a conference paper at ICLR 2025
DQN [200, 200]
DQN [50, 50]
AdaDQN
RandDQN AdaDQN-max
DQN [100, 100]
DQN [25, 25]
AdaDQN, ED = 0
1QM Retur
200
0
200
0
2
3
4
2
3
Env steps
le5
Env steps
1e5
Figure 2: On-the-fly architecture selection on Lunar Lander. All architectures contain two hidden layers. The number of neurons in each layer is indicated in the legend. Left: AdaDQN yields a better AUC than every DQN run. Right: Ablation on the behavioral policy and on the strategy to select the target network used to compute the target. Each version of AdaDQN uses the presented architectures. The strategy presented in Equation (4) outperforms the other considered strategies.
of the best achieved performance by the number of individual trials, as advocated by Franke et al. (2021). We also report the average performance over the individual runs as the performance that a random search would obtain in expectation. Then, we evaluate our method in a second setting for which the hyperparameter space infinite. This setting corresponds the more general case where the RL practitioner has to select hyperparameters for new environment, having limited in- tuitions for most hyperparameters. We compare AdaQN to SEARL (Franke et al. (2021) presented in Section 3), which is specifically designed for this setting, DEHB (Awad et al., 2021), which is a black box optimizer that has been shown to be effective for RI. (Eimer et al., 2023), and random search, which remains very challenging baseline for hyperparameter optimization. In both settings, the remaining hyperparameters are kept unchanged. Appendix B describes all the hyperparameters used for the experiments along with the description of the environment setups and evaluation proto- cols. The code is based on the Stable Baselines implementation (Raffin et al., 2021) and Dopamine RL (Castro et al., 2018)2.
Performance metric. As recommended by Agarwal et al. (2021), we show the interquartile mean (IQM) along with shaded regions showing pointwise 95% percentile stratified bootstrap confidence intervals. IQM trades off the mean and the median where the tail of the score distribution is removed on both sides consider only 50% of the runs. We argue that the final score is not enough to properly compare RL algorithms since methods that show higher initial performances are better suited for real-world experiments compared to methods that only show higher performances later during training. This is why we also analyze the performances with the Area Under the Curve (AUC) that computes the integral of the IQM along the training. We also report the pentt-performig seed to analyze the robustness of AdaQN w.r.t. stochasticity and highlight AdaQN's 's stability against hyperparameter changes. We use 20 seeds for Lunar Lander (Brockman et al., 2016), 9 seeds for MuJoCo (Todorov et al., 2012), and 5 seeds for Atari (Bellemare et al., 2013).
5.1 A PROOF OF CONCEPT
We consider 4 different architectures for AdaDQN and compare their performances to the individ- ual runs in Figure 2 (left). The architectures are composed of two hidden layers, with the number of neurons indicated in the legend. Interestingly, AdaDQN outperforms the best individual archi- tecture, meaning that by selecting different architectures during the training. AdaDQN better copes with the non-stationarities of the optimization procedure. Figure 2 (right) shows the performances for AdaDQN along with version of AdaQN where c = O during the training (A (AdaDQN c 0), i.e. actions are always sampled for the online network that was chosen for computing the target. This variant of AdaQN performs similarly to the best individual run but underperforms compared to AdaDQN because it suffers from passive learning as explained in Section 4. Additionally, we eval- uate version where the target network is selected randomly from the set of online networks, calling this variant RandDQN. This version is similar to the way the target is computed in REDQ (Chen et al., 2020). While sampling uniformly from similar agents yields better performance, shown in Chen et al. (2020), this strategy suffers when one agent is not performing well. In our case, DQN [25, 25] harms the overall performance. Finally, taking the maximum instead of the minimum to select the target (AdaDQN-max) performs as badly as the worst available agent (i.e., DQN [25, 25)).
Thhe code is available in the supplementary material and will be made open source upon acceptance.
6

---

Published as a conference paper at ICLR 2025
DQQ [200, 200]
DON [100, 100]
IXQN [50, 50j
DQN [25, 25]
AdaDON
AdaDQN, Eb =
0
AdaDON- -mx max
Action Target
2
0
3 Env steps
2
4
3
Env steps
lc5
le5 Env steps
1e5
Figure 3: Distribution of the selected hyperparameters for the target network (top) and for the be- havioral policy (bottom) across all seeds. Left: AdaDQN mainly selects the hyperparameter that performs best when evaluated individually. Middle: AdaDQN with e O also focuses on the best individual architecture but does not use all available networks for sampling actions, which lowers its performance. Right: AdaDQN-max is version of AdaDQN where the minimum operator is replaced by the maximum operator for selecting the following target network.
AdaSAC
Random search
Grid search
RandSAC
Worst seed AdaSAC
Worst seed Random search
Worst seed - Grid search
1QM SAC Norm return
best
1.25
1.00
0.75
0.50
0.25
Worst
10
10
10'
0
2
Env Steps
Env Steps
le5
Figure 4: On-the-fly hyperparameter selection on MuJoCo. The 16 sets of hyperparameters are the elements of the Cartesian product between the learning rates {0.0005, 0.001), the optimizers {Adam, RMSProp), the critic's architectures {[256, 256], [512, 512]} and the activation functions [ReLU, Sigmoid). Left: AdaSAC is more sample-efficient than random search and grid search. Right: AdaSAC yields a better AUC than every SAC run while having greater final score than 13 out of 16 SAC runs. The shading of the dashed lines indicates their ranking for the AUC metric.
In Figure 3 (top), we analyze the distribution of the selected hyperparameters for the target network across all seeds along the training. Importantly, AdaDQN does not always select the architecture that performs the best when trained individually, the network DQN [100, 100] is sometimes chosen in- stead of DQN [200, 200]. On the right, AdaDQN-ma max mainly selects DQN [25, 25), which explains its poor performance. The gap in performance between AdaDQN and AdaDQN with e, 0 can be understood by the bottom row of the figure. It shows the distribution of the selected hyperparameters for sampling the actions. Setting e forces the networks not selected for computing the target to learn passively (the top and bottom bar plots are the same for AdaDQN with E = 0). On the contrary, AdaDQN benefits from a better exploration at the beginning of the training (the bottom bar plot is more diverse than the top one at the beginning of the training).
5.2 CONTINUOUS CONTROL: MUJOCO ENVIRONMENTS
We evaluate AdaSAC for a wider choice of hyperparameters on MuJoCo environments. We con- sider selecting from 16 sets of hyperparameters. Those sets form the Cartesian product between the learning rates {0.0005. 0.001]. the optimizers (Adam, RMSProp). the critic's architectures ([256, 256), [512, 512]} and the activation functions [ReL.U, Sigmoid). This sets K to 2 16. The values of the hyperparameters were chosen to be representative of common choices made when tuning by hand a SAC agent. In Figure 4 (left), we compare AdaSAC with a grid search per- formed on the 16 set of hyperparameters. AdaSAC is an order of magnitude more sample efficient than grid search and outperforms random search all along the training. Notably, AdaSAC's worst- performing seed performs on par with the random search approach while greatly outperforming the worst- persst-erforming seed of the grid search. In Figure 4 (right), we show AdaSAC's performance along with the individual performance of each set of hyperparameters. AdaSAC yields the highest.
7

---

Published as a conference paper at ICLR 2025
AdaSAC
RandSAC
IQM rettrn
1e3
Hopper
le3
Ant
1.0
le4
HallCheetah
0.5
Targel
Walker2d
Humanoid
le5
HumanoidStandup
1QM return
2.5
1.0
Target
Worst
3 Env Steps
2
2
3
4
le5
Env Steps
le5
Eny Steps
Figure 5: Per environment IQM return when AdaSAC and RandSAC select from the 16 sets of hyperparameters described in Figure 4. Below each performance plot, bar plot presents the dis- tribution of hyperparameters selected for the target network across all seeds. AdaSAC outperforms RandSAC and most individual run by designing non-trivial hyperparameter schedules.
AUC metric while outperforming 13 out of the 16 individual runs in terms of final performance. Interestingly, AdaSAC reaches the performance of vanilla SAC in less than half of the samples. We also show the performance of RandSAC, for which random target network is selected to compute the target instead of following the strategy presented Equation (5). For clarity, the labels the 16 individual runs
shown. They are available in Figure 11 of Appendix D.I.
We now analyze which set of hyperparameters AdaSAC selects to compute the target. For each envi- ronment, we show in Figure the performances of each hyperparameter when trained individually, AdaSAC and RandSAC performances. Below each performance plot, we report the distribution of hyperparameterss selected to compute the target across all seeds. Overall, the selected networks are changing during the training, which indicates that the loss is not always minimized by the same set of hyperparameters. This supports the idea of selecting the target network based on Equation (4). Fur- thermore, the selected networks are not the same across the different environments. This shows that AdaSAC designs non-trivial hyperparameter schedules that cannot be handcrafted. On Humanoid, selecting the network corresponding to the best-performing agent leads to similar performances to the best rest-performing agents when trained individually. On HumanoidStandup. the selection strategy avoids selecting the worst-performing agents. This is why AdaSAC outperforms RandSAC, which blindly selects agents. HalfCheetah is the only environment where RandSAC slightly outperforms AdaSAC. AdaSAC still yields better final performance than of the individual runs. Finally, on Hopper and Walker2d, AdaSAC outperforms every individual run for the AUC metric by mainly selecting hyperparameters that not performing well when trained individually. The ability of AdaSAC to adapt the hyperparameters at each target update enables it to better fit the targets, which yields better performances.
Ablations. We evaluate AdaSAC against random search and grid search in 4 different scenarios where only one hyperparameter changes. The results presented Appendices D.2, D.3, D.4, and D.5 show that AdaSAC is robust to variations of the hyperparameters. We first give 4 architectures of different capacities as input. Then, we run AdaSAC with 3 different learning rates, and we also another study where AdaSAC is given 3 different optimizers. In each case, AdaSAC matches the best-performing individual runs while successfully ignoring the worst-performing sets of hy- perparameters. Finally, we evaluate AdaSAC with 3 different activation functions given input. This time, AdaSAC outperforms 2 out of the individual runs. Interestingly, RandSAC seems to suffer more in that setting by only outperforming the worst-performing individual run. This further demonstrates that the strategy presented in Equation (4) is effective.
5.3 VISION-BASE CONTROL: ATART 2600
We provide an additional experiment on 3 games of the Atari benchmark. Similarly to the ablation study made the MuloCo benchmark, conclude, in light of Figure that AdaDQN performs
8

---

Published as a conference paper at ICLR 2025
1QM Human Norm Score
Activation function
Architecture
Adam's epsilon
AdaDON
Tank:
AdIDON
AdaDoN
1.56-6
ReLU
Sigmoid
[32, 64, 64, 512)
[66, 16, 16, 714
Eadiro 1.5c4 Exdami
1.308
0.5
0.0
-0.5
0
Target
10
20
30
40
10
20
30
40
10
20
30
40
Num Frames (in millions)
Num Frames (in millions)
Num Frames (in millions)
Figure 6: On-the-fly fly hyperparameter selection on 3 Atari games. AdaDQN performs similarly to the best individual hyperparameters when selecting from activation functions (left), architectures (middle), or 3 different values of Adam's epsilon parameter (right). Below each performance plot, bar plot presents the distribution of the hyperparameters selected for computing the target.
AdaDoN
DEHB
High larget freq High HP froq
SEARL-
SEARL- min
Random search
10M Human Norn Score
1.0
Budget: 40M
0.58
Low larget freq fiign tie fres
High
langet fro Low HP heoo
0.14
0.5
0.0
40
80
120
160
Low target freq Low HP froq
Num Frames (in millions)
Figure 7: On-the-fly hyperparameter selection on Atari games. Each method has to find well- performing hyperparameters in very challenging hyperparameter space composed of 18 activation functions, 24 optimizers, 4 losses, a wide range of learning rates, and fully parameterizable archi- tectures. Left: AdaDQN outperforms SEARL, random search, and grid search in the limited budget of 40M frames and keeps increasing its performance afterward. Right: AdaDQN is robust to hyper- parameter changes. Each axis of the radar plot shows the performances at 40M frames.
par with the pesf-perfogming hyperparameter when the space of hyperparameters is composed of dif- ferent activation functions (left), architectures (middle) or different values of Adam's 's epsilon param- eter (right). This means that random search and grid search underperform compared to AdaDQN. Importantly, the common hyperparameters used for DQN (in orange) are not the best- ones in all settings, which motivates using Auto RL. Below each performance plot, we report the distribution of the hyperparameters selected for computing the target. Remarkably, AdaDQN never selects Sigmoid activations, which perform poorly when evaluated individually. Interestingly, for the ablation study the architecture and Adam's 's epsilon parameter, AdaDQN uses all available hy- perparameters when they perform equally well. We note that, despite not being often tuned, Adam's epsilon parameter plays a role in yielding high returns observed in Seando-Ceron et al. (2024). Figure 20 reports the decomposition of Figure into the considered Atari games.
5.4 INFINITE HYPERPARAMETER SPACES
We now focus on a setting in which the space of hyperparameters infinite. We design a wide space of byperparameters to come closer to the goal of removing the need for prior knowledge about the algorithm and environment. The hyperparameter space, presented in Table 4, comprises 18 acti- vation functions, 24 optimizers, 4 losses, a wide range of learning rates, and fully parameterizable architectures. We let SEARL and AdaDQN train neural networks in parallel. Figure shows how AdaDQN and SEARL differ. To generalize AdaQN to infinite hyperparameter spaces, we replace the RL training step in SEARL's pipeline with the one described in Section 4, and the approximation errors used as fitness values instead of the online evaluation of each network. The exploitation and exploration steps remain identical. We fix a budget of 40M frames on 6 games of the Atari benchmark. We reduce the budget 30M frames for random search favor more individual trials,

---

Published as a conference paper at ICLR 2025
AdaDQN
Meta-Gradiien DQN
1QM IQM Human Homan Norm Score
BattleZZone
Freeway
0.995
1.000
1.0
0.50
0.995
0.25
0.990
0.5
0.990
0.00
0.985
0.0
0.985
20
30
40
20
40
20
30
10
20
40
Num Frames (in millions)
Figure 8: On-the-fly discount factor selection on 2 Atari games. AdaDQN and Meta-gradient DQN perform similarly. On the right of each performance plot, we report the evolution of the discount factor selected by AdaDQN and Meta-gradient DQN.
in total, as the hyperparameter space is large. Finally, we let DEHB benefit from its multi-fidelity component by allowing varying budget from 20M to 40M frames. We report the maximum return observed since the beginning of the training for each seed of random search and DEHB. As every report the maximum and minimum return observed at each
network is evaluated in SEARL, evaluation step (SEARL- - max and SEARL- min).
In Figure 7 (left), at 40M frames, AdaDQN outperforms SEARL and matches the end performance of random search and DEHB. When trained for longer, AdaDQN continues to increase its perfor- mance and outperforms random search and DEHB before reaching 100M frames. We argue that SEARL's low performance is largely explained by the fact that many environment iterations are wasted by poorly performing networks (see SEARL min). which also affects the other networks as the replay buffer is shared among them. Instead, AdaDQN only uses the networks that have a low training loss. Random search and DEHB usually require many environment interactions to perform well. We believe that having wide hyperparameter space exacerbates this phenomenon. Figure 7 (right) highlights AdaDQN's robustness w.r.t. its hyperparameters. Notably, AdaDQN reaches a higher IQM return than SEARL in the 4 considered settings at 40M frames. Figure 21 and 22 show the individual game performances. Figure 23 reports the evolution of the hyperparameters during AdaDQN's training for one seed on Pong. Interestingly, the performances increase (green back- ground) when the network trained on the Huber loss with ReLU activations and Adam optimizer is selected as target network.
AdaQN can also be used to select the target hyperparameters. We propose to select from differ- ent discount factors sampled in [0.985, 0.995]. We consider K 5 online networks and com- pare AdaDQN against Meta-gradient reinforcement learning (MGRL, Xu et al. (2018)). The cross- validated discount factor of MGRL is set at 0.99, and the same discount factor considered when selecting the next target network in AdaQN. Figure 8 shows that the two approaches reach similar performances on 2 Atari games. Nevertheless, we recall that AdaQN differs from MGRL in many points (see Section 3). Crucially, it is not limited to continuous and differentiable hyperparameters.
DISCUSSION AND CONCLUSION
We have introduced Adaptive Q-Network (AdaQN), Q-networks w.r.t. a shared target, selected as the target network corresponding to the most accurate online network. We demonstrated that AdaQN is theoretically sound and we devised its algorithmic implementation. Our experiments show that AdaQN outperforms competitive AutoRL baselines in terms of sample efficiency, final performance, and robustness.
method that trains an ensemble of diverse
Limitations. Our work focuses on the agent's hyperparameters: thus, environment's properties, e.g., reward, cannot be handled by AdaQN. Nevertheless, an off-the-shelf AutoRL algorithm can be combined with AdaQN to handle these properties. Moreover, since AdaQN considers multiple Q- functions in the loss, the training time and memory requirements increase. We provide an analysis in Appendix C. to show that this increase remains reasonable compared to the gain in performance and sample-efficiency. We stress that, in theory, the computations can be parallelized so that the adaptive version of an algorithm requires the same amount of time as its original algorithm. The additional "for loop" in Line 9 of Algorithm and the one in Line 7 of Algorithm can be run in parallel as long enough parallel processing capacity is available, as it is common in modern GPUs. Finally, instead of selecting one Q- Q-function for computing the target, future works could consider a mixture of Q Q-functions learned with different hyperparameters presented in Seyde et al. (2022).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was funded by the German Federal Ministry of Education and Research (BMBF) (Project: 011S22078). This work was also funded by Hessian.ai through the project 'The Third Wave of Ar- tificial Intelligence - 3AI by the Ministry for Science and Arts of the state of Hessen, by the grant "Einrichtung eines Labors des Deutschen Forschungszentrum für Künstliche Intelligenz (DFKI) an der Technischen Universität Darmstadt", and by the Hessian Ministry of Higher Education, Re- search, Science and the Arts (HMWK).
CARBON IMPACT
As recommended by Lannelongue Inouye (2023), we used Grrennlgors (Lannelongue al., 2021) and ML CO2 Impact (Lacoste et al., 2019) to compute the carbon emission related to the production of the electricity used for the computations energy used to generate the figures presented in this work and ignore the energy used for preliminary studies. The estimations vary between 1.30 and 1.53 tonnes of CO2 equivalent. As a reminder, the Intergovernmental Panel Climate Change advocates carbon budget of 2 tonnes of CO2 equivalent per year per person.
our experiments. We only consider the
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Infor- mation Processing Systems, 2021.
Matthew Aitchison. Penny Sweetser, and Marcus Hutter. Atari-5:
Distilling the arcade learning environment down to five games. In International Conference on Machine Learning, 2023.
Marcin Andrychowiez, Anton Raichuk, Piotr Stañczyk, Manu Orsini, Sertan Girgin, Raphaäl Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2020.
Noor Awad, Neeratyoy Mallik, and Frank Hutter. Dehb: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In International Joint Conference on Artificial Intelligence, 2021.
Adriá Puigdoménech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, 2020.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning, 1995.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. In Journal of artificial intelligence research, 2013.
James Bergstra and Yoshua Bengio. Random search for hyper- parameter optimization. In Journal of Machine Learning Research, 2012.
A Bhatt, D Palenicek, B Belousov, Max Argus. Artemij Amiranashvili, Thomas Brox, and J Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In International Conference on Learning Representations, 2024.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. In arXiv preprint 1606.v1600.01550 2016.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. http://arrivoor//ab/////11000000 2018.
11

---

Published as a conference paper at ICLR 2025
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W Ross. Randomized ensembled double q- learning: Learning fast without a model. International Conference on Learning Represen- tations, 2020.
Yutian Chen, Aja Huang, Ziyu Wang, loannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in alphago. In arXiv preprint arXiv:18/2006855, 2018 8.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International Conference on Machine Learning, 2018.
Carlo D' Eramo and Georgia Chalvatzaki. Prioritized sampling with intrinsic motivation in multi- task reinforcement learning. In International Joint Conference on Neural Networks, 2022.
Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. In International Conference on Machine Learning, 2023.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International Conference on Learning Representations, 2019.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale. In International Conference on Machine Learning. 2018.
Amir-massoud Farahmand. Regularization in reinforcement learning. PhD) thesis, University of Alberta, 2011.
Amir-massoud Farahmand and Csaba Szepesvári. Model selection in reinforcement learning. In Machine learning, 2011.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. aodel-agnostic meta- meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satin- der Singh. Bootstrapped meta-learning. In International Conference Learning Representa- tions, 2021.
Jörg K.H. Franke, Gregor Koehler, André Biedenkapp, and Frank Hutter. Sample-efficient auto- mated deep reinforcement learning. International Conference on Learning Representattons, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning. 2018
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforeement learning that matters. In Association for the Advancement of Artificial Intel- ligence, 2018.
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning Meth- ods, Systems, Challenges. Springer, 2019.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, JW Böhmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. International Conference on Learning Representations, 2021.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Ko- ray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population-basee reinforcement learning. In Science, 2019.
Maxwell Elliot Jaderberg, Wojciech Czarnccki, Timothy Frederick Goldie Green, and Valentin Clement Dalibard. Population based training of neural networks. In US Patent, 2023.
12

---

Published as a conference paper at ICLR 2025
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. In National Academy of sciences, 2017.
Pascal Klink, Carlo D' Eramo, Jan R Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint 19XX:v/9900090000 2019.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Representations, 2020.
Loic Lannelongue and Michael Inouye. Carbon footprint estimation for computational research. Nature Reviews Methods Primers, 2023.
Loic Lannelongue, Jason Grealey, and Michael Inouye. Green algorithms: quantifying the carbon footprint of computation. Advanced Science, 2021.
Jonathan N Lee. George Tucker, Ofir Nachum, Bo Dai, and Emma Brunskill. Oracle inequalities for model selection in offline reinforcement learning. In Advances Neural Information Processing Systems, 2022.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In arXiv preprint
2015.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018.
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. In Journal of artificial intelligence research, 2018.
A. Rupam Mahmood, Dmytro Korenkevych, Gautham Vasan, William Ma, and James Bergstra. Benchmarking reinforcement learning algorithms real-world robots. In Conference Robot Learning, 2018
Brad Miller, David E Goldberg, et al. Genetic algorithms, tournament selection, and the effects of noise. Complex systems, 1995.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. In Nature, 2015.
Aditya Mohan, Carolin Benjamins, Konrad Wienecke, Alexander Dockhor, and Marius Lindauer. Autorl hyperparameter landscapes. International Conference on Automated Machine Learning, 2023.
Vu Nguyen, Sebastian Schulze, and Michael A Osborne. Bayesian optimization for iterative learn- ing. In Advances in Neural Information Processing Systems, 2020.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning, In International Conference on Machine Learning. 2022.
Johan Obando-Cerow João GM Araújo, Aaron Courville, and Pablo Samuel Castro. On the consis- tency of hyper-parametee selection in value-based deep reinforcement learning. In Reinforcement Learning Conference, 2024.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
13

---

Published as a conference paper at ICLR 2025
Jack Holder, Raghu Rajan, Xingyou Song, André Biedenkapp. Yingjie Miao, Theresa Eimer, Baohe Zhang, Vu Nguyen, Roberto Calandra, Aleksandra Faust, et al. Automated reinforeement learning (autorl): A survey and open problems. In Journal of Artificial Intelligence Research, 2022.
Martin L Puterman. Markov decision processes. Handbooks in Operations Research and Manage- ment Science, 1990.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dor- mann. btable-baselines:: Reliable reinforcement learning implementations. In Journal of Ma- chine Learning Reseurch, 2021.
Carlos Riquelme, Hugo Penedones, Damien Vincent, Hartmut Maennel, Sylvain Gelly, Timothy A Mann, Andre Barreto, and Gergely Neu. Adaptive temporal-dite -iffeeeeeee learning for policy evalua- tion with per-state uncertainty estimates. In Advances in Neural Information Processing Systems, 2019.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In arXiv preprint arXiv:1511.05952, 2015.
Tom Schaul, Diana Borsa, David Ding, David Szepesvari, Georg Ostrovski, Will Dabney, and Simon Osindero. Adapting behaviour for learning progress. In arXiv preprint 192iv: 2019.
Tim Seyde, Wilko Schwarting, Igor Gilitschenski, Markus Wulfmeier, and Daniela Rus. Strength through diversity: Robust behavior learning via mixture policies. In Conference Robot Learn- ing, 2022.
Gresa Shala, André Biedenkapp. Frank Hutter, and Josif Grabocka. Gray-box gaussian processes for automated reinforcement learning. In International Conference on Learning Representations, 2022.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. In Nature, 2017.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, 2023.
Kenneth O. Stanley, David B. D' Ambbosio, and Jason Gauci. A hypercube-based encoding for evolving large-scale neural networks. In Artificial Life, 2009.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 1998.
Richard Sutton and Satinder P Singh. On step-size and bias in temporal-difference learning. In Yale Workshop on Adaptive and Learning Systems, 1994.
Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical con- siderations for healthcare settings. In Machine Learning for Healthcare Conference, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
Théo Vincent. Daniel Palenicek, Boris Belousov, Jan Peters, and Carlo D'Eramo. Iterated q- network: Beyond one-step bellman updates in deep reinforcement learning. Transactions on Machine Learning Research, 2025.
Martha White and Adam White. greedy approach to adapting the trace parameter for temporal difference learning, In International Conference Autonomous Agents and Multiagent Systems, 2016.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in Neural Information Processing Systems, 2018.
14

---

Published as a conference paper at ICLR 2025
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. Advances in Neural Informa- tion Processing Systems, 2020.
Joshua P Zitovsky, Daniel De Marchi, Rishabh Agarwal, and Michael Rene Kosorok. Revisiting bellman errors for offline model selection. In International Conference on Machine Learning, 2023.
15

---

Published as a conference paper at ICLR 2025
A THEOREM STATEMENTS AND PROOFS
Theorem 3.4 from Farahmand (2011), Let N N", and p. V two probability measures on x A. For any sequence (0:) E @N+1 where Ry depends the reward function and the discount factor, we have
112" 1 I CN..R + inf F(r; ((r:NN,pn) re(0.1)
(En
En af
Qū, 11
where By, ((r:N,,P?)) and (as) do not depend on (0.) TN is greedy policy com- puted from Qix-
Theorem 4.1. Let (0*) k=1 E @K and 0 θ be vectors of parameters representing K + 1 Q- functions. Let D (,,,a,r,s') be a set of samples. Let v be the distribution represented by the state-action pairs present in D. We note Dz,a {(r, 8')) a, E D), (s, D.
If for every state-action pair (s,a) D, ((r,s [Q)(s,)) rQq(s,a). then have
arg min kE(I.....K
[[Q] - Q0 arg min kE(l.... (saa)s))
Σ
Lev(0 0,,,ar,,8))
Proof. Let (Oh) F=1 E O and 0 E θ be vectors of parameters representing K+1 1 Q-functions. Let D = ((s, a, s7 be a set samples. Let v be the distribution of the state-action pairs present in D. In this proof, we note I' as Frr. to stress its dependency on the reward and the next state s'. For every state-action pair (s, a) in D, we define the Da,a {(r, s') (s, a, r')') E DD and assume that ((r,s')>,,, [T r,'s' Q0(s, a)] = ['Qa(s, a). Additionally, we note M the cardinality of D, M the cardinality of De.a and D the set of unique state-action pairs in D. We take & E {1...., K} and write
Σ 1, (s.D,
Σ (s.D
Qo -
Σ (sa.))))
-Q -Qm(ss.)))
Σ
(frr rQo(ss))))
(s.a.,a))E
Σ
(TQa(s,a) - Qs,
(s.),;,"))
+2
Σ
(Tno (ono))(s,a) - TQo (TQa(s,a) - Qo (s,
The second last equation is obtained by introducing the term TQ a) and removing it. The last equation is obtained by developing the previous squared term. Now, we study each of the three terms:
(Fr. -QQ(s,))) 2 independent 0 Eisans (To
E(sana'ee (TQp(),a) (8, a))² equal M ||TQa Qo+ 112, by definition of
16

---

Published as a conference paper at ICLR 2025
Σ (PreQo(t.)- rQQ(sa,)) (TQa(s,a)= Qon (s,
Σ (Ma)ED
Σ (1Q0(a))))- (TQQ(ss,a) Qo)(s)))) -(o =0 ((na')EDaa
since, for every (s D.
Σ -QQ((ss) = Ma,a Qods -rQo(ssa)))))) =
(r,')ED,,a
the last equality holds from the assumption.
Thus, we have
Σ
CON (0,,,,,, constant w.r.t 6² + M x |[[Qq - Qg+ 11,,
(s,0,
This is why
arg min RE (e(....,)) (s.an,e))
E
LoN(0"10,s,,,))
arg min ke Ae(.....
[[[Qa - Qaa Qarl
A.1 CONVERGENCE OF ADAQN
In this section, we consider tabular version of AdaQN in a finite MDP where the discount fac- tor y < 1. Our goal is to prove the convergence of AdaQN to the optimal action-value function by adapting the proof of Theorem Q-learning framework (Lan et al., 2020) does not comprise AdaQN. Indeed. while the target com- putation of the Generalized Q-learning framework (Equation (3) in Lan et al. (2020)) only depends on a fixed history length, AdaQN's target computation (Equation (4)) depends on tb, the index of the selected network used for computing rithm. This is why, in the following, we first present the modifications brought to the Generalized Q-learning framework before proving the convergence.
Lan al. (2020). Without modification, the Generalized
target, which is defined from the entire history of the algo-
We color in blue the modifications brought to the Generalized Q-learning framework. Using the same notation as in Equations (15) and (16) in Lan et al. (2020), for a state s, an action a, and at timestep t, the new update rule is
Q' 1) O' + a' (Q((), (00) (',a()) (t) + (t) Vi (I,.,,N, (6)
where (1)) E (1,.., N) is F(t) and
(Q Wa,a (t) = -s.a -iirsal + (o ((..a) (t) - ElQ GQ. (+)) ((,aa)
(7)
We recall that in Lan al. (2020), Q TRmnNK where is the cardinality of S, the cardinality of A, N' is the population size, and K is the history length where Q (t) is the component corre- sponding to the 7th element of the population at timestep / 1- for state and action F(() is the o-algebra representing the history of the algorithm during the first t iterations. a²,a(t) is the step size and f(s, a) a random successor state starting from state after exccuting action a. We define, for E (1,.., NJ, F..a (Q, E[rs.a] + +E(Q (e.aa). where QGQ, G(Q., i).
For AdaQN, (t) represents the index of the selected target at timestep t. Following Equa- tion (4), (t) = arg min, [|E[ra.a] + E0 Q2 with (0() = 0, and for i (1,-, N), G(Q,, i) max, Q It is clear that (i) is m(aauuuurr..
((s.a)
17

---

Published as a conference paper at ICLR 2025
We now verify that Assumptions 1, 3, 4, and 6 in Lan et al. (2020), needed to prove the con- vergence according to Theorem 5 in Lan et al. (2020), hold for AdaQN. Assumption holds as we have (G),,) c) - ,() S maxa,i. Q2 - Q²" As- sumptions 3, 4, and 6 also hold as the modification of the target computation does not influence their validity. We point out that the constraint that (t) is F())mmeauuarble is used to verify that ws (t) F(t + meaasuraee for all state s and action a. This validates Assumption 4 (zi). Therefore, by defining the step sizes such that Assumption 2 holds, AdaQN converges to the optimal action-value function with probability
VQ.Q' E BINK ,,,' [L,..,N}, J.
B ALGORIHTMM AND HYPERPARAMETERS
Algorithm 2 Adaptive Soft Actor-Critic (AdaSAC). Modifications SAC are marked in purple.
1: Initialize the policy parameters é, K online parameters (0*) and an empty replay buffer D. For k 1..... K, set the target parameters 0* + 0* and the cumulative losses Ly = 0. Set and the the indices to be selected for computing the target.
2: repeat
3: Take action a, ~ To (-1ss)) Observe reward Te, next state st+1: D. + DUE((s)) aellss (6.)))))))
4: for UTD updates do
5: Sample mini-batch B = {(,,,,,,,)) from D.
6: Compute the shared target
y x+ (v))))))))))
min
( a aolog log' where a' aa(-ss).
7:
for k = 1,.... K do
8:
Compute the loss w.r.t 0h, -(sanses) (y Qo)(s,)))²
9:
Update LA < (1-1))Le + TLŌN-
10:
Update 0² using its specific optimizer and learning rate from V LōN-
11:
Update the target networks with 0* +- -00 + (1 (1-7)) (1-r)0k.
12:
Set t/1 and the to be the indexes of the 2 lowest values of
13:
Set (169, 165) DistinctU/(,,... K} w.p e and (vf, v/2)) (1), th2) w.p 1 - co.
14:
Update á with gradient ascent using the loss
min Qa (s, log (alss), a no((s)) kE(v9.01
B. EXPERIMENTAL SETUP
All environments are generated from the Gymansium library (Brockman et al., 2016). For the Atari experiments, we build our codebase on Vincent et al. (2025) which mimics the implementation choices made in Dopamine RL (Castro et al., 2018). Dopamine RL follows the training and eval- uation protocols recommended by Machado et al. (2018). Namely, we use the game over signal to terminate an episode instead of the life signal. The input given the neural network is concatena- tion of 4 frames in grayscale of dimension 84 by 84. To get a new frame, we sample 4 frames from the Gym environment configured with no frame-skip, and apply a max pooling operation on the 2 last grayscale frames. We use sticky actions to make the environment stochastic (with 0.25). The performance the one obtained during training. We consider a different behavioral policy for the experiment with an infinite hyperparameter space. Instead of taking an e-greedy policy param- eterized by 6 as explained in Section 4.1, we sample each action from probability distribution that is inversely proportional to the current loss value. That way, the problem of passive learning is further mitigated since the second or third most accurate networks will have a higher probability of being selected to sample actions. Importantly, when AdaDQN selects between different losses, the comparison made between the values the L2 losses even if the backpropagation is performed with another loss. The 3 Atari games in the experiments of Section 5.3 are chosen from the 5 games presented in Aitchison et al. (2023). The 6 Atari games in the experiments Section 5.4 are chosen from Franke al. (2021).
18

---

Published as a conference paper at ICLR 2025
AdaDQN
Learn all Qi to be closer to Q
After M
Compute the approximation error fj of cach Q3 w.r.t. Q
Every T steps Q Q where 1o = argmin [rQ Q,il
steps
Exploit and explore using fi
Initialize all Qi
Learn each Qi to be closer Qi
After
Compute the performance of cach Qi
steps
Every T steps Q
SEARL
Figure 9: AdaDQN and SEARL pipelines. AdaDQN and SEARL. share the same exploitation and exploration strategies. They only differ the way the Q-functions are trained and the nature of the fitness value ft. In both algorithms, the exploitation and exploration phase is performed every M steps. It referred to as the hyperparameter frequency (HP freq) in Figures and 22.
AdaQN SEARL comparison. The exploitation strategy of SEARL and AdaDQN is based on tournament selection with elitism (Miller et al., 1995). The tournament comprises K steps for population of K networks. At each step, 3 networks are sampled uniformly without replacement, and the network with the best fitness value Sa qualified to populate the next generation. The last spot is reserved for the network with the best fitness value across the entire population. Then, every network present more than twice in the population undergoes a mutation operation. This is the exploration phase. One mutation operation out of the 5 operations activation function change, loss change, optimizer change, and learning rate change) is chosen with equal probability. When the architecture is changed,
consider (architecture change,
of the following operations is selected {#1 CNN layer, +4 CNN channels, +1 kernel size, +1 CNN stride, #1 MLP layer, 16 MLP neurons) 0.4
with probability [0.1, 3 3 3. 0.11 0.4]. Those probabilities are not tuned. They are by hand so that the number of layers of the CNN and MLP do not change too often. If the hyperparameter Table 4, the value is clipped to the boundary. In every
goes outside the search space described case, the network weights are preserved as much as possible. This means that when the architecture changes, only the new weights that were not present before the mutation operation are generated. We choose reset the optimizer each time network undergoes genetic modification. Instead of an elitism mechanism, we investigated, on Lunar Lander, a truncation mechanism (Jaderberg et al., 2023) where the 20% most accurate networks are selected for the next generation. The performances are similar to the ones obtained with the elitism mechanism. We also tried to reset the weights after each generation. This did not lead to performance improvements. Therefore, kept the proposed pipeline as close to SEARL as possible.
SEARL evaluates each network's return empirically estimate its performance. We the min- imum number of environment steps to evaluate each network to M divided by the number of net- works in the population that least M new samples are collected in the replay buffer. This also synchronizes the frequency at which the hyperparameters are updated with AdaDQN. SEARL network weights are updated every 4 training step, similarly AdaDON (G in Table 4).
AdaQN Meta- reeta-gradeet RL comparison. The experiment comparing AdaDQN to Meta- deta-gradient DQN made with the default hyperparameters of DQN. Each network of AdaDQN is trained with a different discount factor, and the comparison between online networks is made by computing the approximation error parameterized by cross-vallee cross-validated discount factor 7. The optimizer used for the outer loss of Meta-gradient DQN is Adam with a learning rate of 0.001 as in the original paper (Xu al., 2018). Similarly to Xu et al. (2018), we set the size of the discount factor's embedding space to 16.
B.2 EVALUATION PROTOCOL
The reported performance of the individual runs DQN and SAC as well as AdaDQN, AdaDQN with é, 0, RandDQN, AdaDQN-maxx, AdaSAC, RandSAC, and Meta-Gradient DQN the
19

---

Published as a conference paper at ICLR 2025
raw performance obtained during training aggregated over seeds and environments. The reported performances of the other methods require some post prsc-proes steps to account for the way they operate. In the following, for a hyperparameter l and after t environment steps, we note & (t) note K the number
the return obtained during training for a task and seed j. Additionally, of hyperparameters considered for the specific setting. For the finite search space setting, we first compute all possible individual runs (Figure 4 (right)) and, in Figure 4 (left), we derive:
Grid search as the IQM of the best individual hyperparameters across each task at each timestep t. We scale horizontally the IQM the right by the number of individual runs to account for the environment steps taken by all individual runs. Indeed, we first identify the hyperparameter kg (t) with the highest IQM for each task i each timestep t: Ky (t) arg max IQM, (s (t)). Then, we report IQM grid scarch (t x K) = IQM, ( kg (1) (t)).
Random search the empirical average over all individual runs. We approximate the em- pirical average using Monte Carlo sampling for the possible sequence of hyperparameters. This corresponds the expected performance of what a random search would look like if only one individual trial were attempted sequentially.
In the infinite search space setting (Figure 7 (left)), report the following algorithm:
Random search & DEHB: to make random search and DEHB more competitive in this setting, we keep the samples collected by the previous individual trials when a new trial starts. This means that, for each task and each seed, random search and DEHB generate sequence of returns over 180M environment interactions. From this sequence, we compute a running maximum over timesteps. Finally, we compute the IQM over seeds and tasks. In detail, the score used for computing the IQM is maxe 6.j (t').
SEARL: SEARL has the specificity of periodically evaluating every Q-network of its pop- ulation. Each time every Q-network is evaluated,
use the best return to compute the
IQM of SEARL-max (maxRe (I....K 5/ (t)) and the worst return compute the IQM of SEARL-min (minge(l.... 8% (t)).
Table 2: Summary of all fixed hyperparameters for the MuloCo experiments (see Section 5.2).
Table I: Summary of all fixed hyperparameters used for the Lunar Lander experiments (see Sec- tion 5.1).
Shared across all algorithms
Discount factor y
0.99
Shared across all algorithms
Horizon
1000
Discount factor y
0.99
Initial replay buffer size
5000
Horizon
1000
Replay buffer size
106
Initial replay buffer size
1000
Batch Size
256
Replay buffer size
10a
Target update rate
0.005
Batch Size
32
UTD
Target update frequency T
200
Actor's activation function
ReLU
Training frequency G
Actor's learning rate
0.001
Activation function
ReLU
Actor's optimizer
Adam
Learning rate
3 x 10-4
Actor's architecture
256, 256
Optimizer
Adam
Policy delay
Loss
L2
Vanilla SAC
Starting
Critic's activation function
ReLU
Ending
0.01 1000
Critic's 's learning rate
0.001
Linear decay duration of AdaDQN
Critic's optimizer
Adam
Critic's architecture
256, 256
Starting
AdaSAC
Ending 6
0.01 Until end
Starting Cb
Linear decay duration of c
Ending eb
0.01 Until end
Linear decay duration of ea
20

---

Published as a conference paper at ICLR 2025
Table 4: Summary of all hyperparameters used for the Atari experiments with an infinite hyperparameter space (see Section 5.4).
Shared across all algorithms
Discount factor 7
0.99
Horizon
27000
Full action
No
space
Reward
clip(
11
Table 3: Summary of all fixed hyperparame- ters used for the Atari experiments with a fi- nite hyperparameter space (see Section 5.3).
clipping
Initial replay
20000
buffer size
Replay
10c
buffer size
Shared across all algorithms
Batch Size
32
Discount factor 7
0.99
Training
Horizon
27000
4
frequency G
Full action space
No
Reward clipping
clip(-1,11 1)
Starting e
Ending
0.01
Initial replay buffer size
20000
Linear decay
Replay buffer size
106
250000
duration of c
Batch Size
32
Hyperparameter space
Target update frequency T
8000
[CELU, ELU, GELU, Hard Sigmoid, Hard SiLU, Hard tanh, Leaky ReLU, Log-Sigmoid, Loog-Softmax, ReLU, SELU, Sigmoid, SiLU, Sofi-sign, Softmax, Softplus, Standardize, Tanh}
Training frequency G
4
Learning rate
10
Optimizer
Adam
Activation function
Loss
1.2
Starting e
Ending
0.01
Linear decay duration of
250000
Learning rate (in log space)
Vanilla DQN
[10-5 60-6,0 10 10
Activation function
ReLUU
[AdaBelief, AdaDelta,
1.5 10
CAdam
AdaGrad, AdaFactor,
CNN layers
3
Adam, Adamax, Adamax W. AdamW, AMSGrad, Fromage, Lamb, Lars, Lion, Nadam, Nadam W. Noisy SGD, Novograd, Optimistic GD, RAdam, RMSProp, RProp, SGD. SM3, Yogi]
CNN channels
[32, 64, 64
CNN kernel size
18,4 1,
CNN stride
Optimizer
MLP layers
MLP n neurons
512
AdaDQN
Starting
1
Ending ED
0.01 Until end
Linear decay duration of eb
Loss
(Huber, L1, L2, Log cosh
CNN layers
[1,3]
CNN channels
[16. 64
CNN kernel size
[2,
CNN stride
[2,
MLP layers
[0.2]
MLP neurons
[25, 512
Different Settings
High target freq
T 8000
Low target freq
T 4000
High HP freq
M 80000
Low HP freq
40000
21

---

Published as a conference paper at ICLR 2025
C TRAINING TIME AND MEMORY REQUIREMENTS
We address time complexity by reporting the performances obtained in Section 5.2 and 5.4 with the clock time the horizontal axis in Figure 10. On the left, AdaSAC still outperforms grid search but is slower than random search as AdaQN carries out several networks in parallel. We stress that this experiment is designed demonstrate the capabilities of AdaSAC when high number of hyperparameters is considered. When the number of network decreases to 5, AdaQN becomes more competitive against random search Figure 10 (right) testifies. In this case, AdaDQN outperforms all baselines in terms of training time and sample efficiency. We point out that Meta-gradient DQN, evaluated Section 5.4, is 4% slower to run than AdaDQN. Finally, we recall that, in this work, the main focus is on sample-cfficienge as it is the predominant bottleneck in practical scenarios.
Concerning the memory requirements, AdaSAC uses an additional 300Mb from the vRAM for the MuloCo experiments presented in Section 5.2 compared to SAC. AdaDQN uses 200Mb more from the vRAM than random search for the Atari experiments presented in Section 5.4. We argue that those increases remain small compared to the benefit brought by AdaQN.
AdaSAC
Worst seed AdaSAC
AdaDQN
DEHB
Random search
Worst seed Random search
SEARL max
Random search
Grid search
Worst seed
Grid search
SEARL min
1QM SQM Norm Nom return
1QM Human Norm Score
1.0
1.25
1.00
0.75
0.5
SAC
0.50
0.0
0.0
8.0
16.0
24.0
33.0
0.0
5.0
10.0
15.0
20.0
Clock time (in hours)
Clock time (in hours)
Figure 10: Performances observed in Section 5.2 (left) and Section 5.4 (right) according to the train- ing time. When evaluated with a high number of hyperparameters to consider in parallel, AdaQN suffers from slower training time but is still faster than grid search. When the number of hyperpa- rameters is reasonable (z= 5). AdaQN is competitive against all baselines.
D ADDITIONAL EXPERIMENTS
D.1 ON-THE-FLY HYPERPARAMETER SELECTION ON MuJoCo
SAC learning rate: 0.001, optimizer: Adam, architecture: 512, 512, activation fn: ReLU
SAC leurning 0.0005, optimizer: Adam, architecture: 256, 256, activation fn: ReLU
SAC leurning rule: 0.001, optimizer: Adam, architecture: 512, 512, activation fn: Sigmoid
512, activation RcLU SAC learning 0,0005, optimizer: Adam, architecture:
SAC learning rale: 0,001, optimizer: Adam, architecture: 256, 256, activation fn: Sigmoid
SAC learning rate: 0,0005, optimizer: Adam, architecture: 512, 512, activation Sigmoid
SAC learning rate: 0.000S, optimizer: RMSProp, architecture: 256, 256, activation fn: ReLU
SAC leurning rute: 0,0005, optimizer: RMSProp, architecture: 512, 512, activation fn: ReLU
SAC leurning 0.001, optimizer: RMSProp, architecture: 512, $12, activation fn: Sigmoid
SAC lcarning rate: 0,0005, optimizer: Adam, architccture: 256, 256, activation fn: Sigmoid
SAC learning rate: 0.001, optimizer: RMSProp, architecture: 256, 256, activation fn: ReLU
vanilla SAC leamming rate: 0.001. optimizer: Adam, architecture: 256, 256, activation fu: ReLU
SAC learning rate: 0,001, optimizer: RMSProp, architecture: 256, 256, activation fn: Sigmoid
SAC leurning 0.001, optimizer: RMSProp, architecture: 512, 512, activation fn: ReLU
SAC leurning rute: 0.0005, optimizer:
RMSProp, architecture: 512, 512, activation fn: Sigmoid
SAC learning 0.0005, optimizer:
RMSProp, architccture: 256, 256, activation Sigmoid
Figure 11: Exhaustive legend of Figure showing the ranking by AUC of the 16 considered sets of hyperparameters. Interestingly, the ranking does not follow predictable order, as there is no clear ordering between each hyperparameter. Importantly, the ranking changes between MuJoCo environments, justifying the usage of AdaSAC, which adapts the hyperparameter schedule for each environment and each seed.
22

---

Published as a conference paper at ICLR 2025
D.2
ON-THE-FLY ARCHITECTURE SELECTION ON MuJoCo
Random
search
RandSAC
Worst -ddSSC
Worst seed Random search
Worst seed
Grid search
1QM JQMSAC SAC Noom return
1.00
0.75
0.50
0.25
10
10
10
Env Steps
Env Steps
1c5
SAC architecture: 256, 256
SAC architecture: 256, 256, 256, 256
SAC arCrhiitoctur:: 256,
SAC arcchitecture: 256
Figure 12: On-the-fly fly architecture selection on MuJoCo. All architectures contain hidden layers with 256 neurons. The architectures indicated in the legend. Left: AdaSAC is more sample- efficient than grid search. Right: AdaSAC yields similar performances as the best-performing architectures. RandSAC and AdaSAC are par.
AdaSAC
SAC architocture: 256, 256
SAC architocture: 256, 256, 256
RandSAC
SAC architecture: 256
SAC architecture: 256, 256, 256, 256 HalfCheetah
IQM return
le3
Hopper
le3
Ant
1e4
2
0.5
0.0
Targer
IQM return
Walker2d
le3
Humanoid
1.50
le5
HumanoidStandup
2.5
1.25
Target
4
4
4
Env Steps
le5
Env Stops
le5
Env Steps
le5
Figure 13: Per environment return of AdaSAC and RandSAC when 4 different architectures are given as input. Below each performance plot, a bar plot presents the distribution the hyperpa- rameters selected for computing the target across all seeds. AdaSAC effectively ignores the worst- performing architecture (256) while actively selecting the best-performing architecture. Interest- ingly, in HumanoidStanddup, despite not performing well, AdaSAC selects the architecture 256, 256, which is the best-performing seaefoomiing-rrchiccttuee architecture.
23

---

Published as a conference paper at ICLR 2025
D.3
ON-THE-FLY LEARNING RATE SELECTION ON MuJoCo
AdaSAC
Random search
Grid search
RandSAC
Worst seed AdaSAC
Worst seed Random search
Worst seed Grid search
1.00
IQM SAC Nom
0.75
0.50
10*
10
10
Env Steps
Env Steps
le5
SAC learning rate: 0.001
SAC learning rute: 0.0005
SAC learning rate: 0.005
Figure 14: On-the-fly learning rate selection on MuJoCo. The space of hyperparameterss is com-
posed of 3 different learning rates. Left: AdaSAC is more sample-efficient than random search and grid search. Right: AdaSAC yields performances slightly above the best best-performing learning rate. RandSAC and AdaSAC are on par.
AdaSAC
SAC learning rate: 0.001
SAC learning rate: 0.0005
RandSAC
SAC learning rate: 0.005
le3
Hopper
le3
Ant
le4
HalfCheetah
IQM return
1.0
2.5
0.5
1
0.0
Target
le3
Walker2d
1e3
Humanoid
le5
HumanoidStandup
IQM retur
5.0
1.5
2
2.5
1.0
Target
2
5
2
5
2
4
Env Steps
lc5
Env Steps
lc5
Env Steps
lc5
Figure 15: Per environment return of AdaSAC and RandSAC when 3 different learning rates
given as input. Below each performance plot, a bar plot presents the distribution of the hyperpa- rameters selected for computing the target across all seeds. AdaSAC creates custom learning rate schedule for each environment and each seed. Interestingly. this schedule seems to choose a low learning rate the beginning of the training before increasing it later during the training.
24

---

Published as a conference paper at ICLR 2025
D.4
ON-THE-FLY OPTIMIZER SELECTION MuJoCo
AdaSAC
Random search
Grid search
RandSAC
Worst seed AdaSAC
Worst seed Random search
Worst seed Grid search
1.00
IQM SAC Nom
0.75
0.50
10*
10
10
Env Steps
Env Steps
le5
SAC optimizer: Adam
SAC optimizer: Adam
SAC optimizer: RMSProp
Figure 16: On-the-fly optimizer selection on MuJoCo. The space of hyperparameters is com-
posed of 3 different optimizers. Left: AdaSAC is more sample-efficient than random search and grid search. Right: AdaSAC yields similar performances as the best-performing optimizer (AdamW, Loshchilov & Hutter (2018)). RandSAC and AdaSAC are on par.
AdaSAC
SAC optimizer. Adam
SAC optimizer: RMSProp
RandSAC
SAC optimizer. AdamW
le3
Hopper
le3
Ant
1e4
HalfCheetah
IQM return
1.0
2.5
0.5
1
0.0
Target
le3
Walker2d
1e3
Humanoid
le5
HumanoidStandup
IQM retur
5.0
1.5
4
2
2.5
1.0
Target
2
5
2
2
Env Steps
lc5
Env Steps
lc5
Env Steps
lc5
Figure 17: Per environment return of AdaSAC and RandSAC when 3 different optimizers given as input. Below each performance plot, a bar plot presents the distribution of the hyperparameters selected for computing the target across all seeds. AdaSAC effectively ignores RMSProp, which performs poorly in most environments.
25

---

Published as a conference paper at ICLR 2025
D.5
ON-THE-FLY ACTIVATION FUNCTION SELECTION ON MuJoCo
AdaSAC
Random search
Grid search
RandSAC
Worst seed AdaSAC
Worst seed Random search
Worst seed
Grid search
1.00
IQM SAC Nom
0.75
0.50
10*
10
10
Env Steps
Env Steps
le5
SAC activation fn: ReLU SAC activation fr:: Sigmoid
SAC activation fn: Tunh
Figure 18: On-the-fly activation function selection on MuJoCo. The space of hyperparameters is composed of 3 different activation functions. Left: AdaSAC is sample-efficient than random search and grid search. Right: AdaSAC performs slightly below the best-performing activation function. In this setting, RandSAC suffers from the fact that network with Tanh activation functions is selected as the target network one-third of the time.
AdaSAC
SAC activation fn: ReLU
SAC activation fn: Tanh
RandSAC
SAC activation fn: Sigmoid
IQM retrm
le3
Hopper
le3
Ant
le3
HalfCheetah
7.5
5.0
2.5
Target
IQM retur
le3
Walker2d
5.0
le3
Humanoid
1e5
HumanoidStandup
1.5
2-
2.5
1.0
Target
2
Env Steps
le5
Env Steps
lc5
Env Steps
lc5
Figure 19: Per environment return of AdaSAC and RandSAC when different activation functions
given as input. Below each performance plot, bar plot presents the distribution of the hy- perparameters selected for computing the target across all seeds. Remarkably, AdaSAC selects the Sigmoid activation function in environments where this activation function HalfCheetah, and Walker2d), while it does not select the Sigmoid activation function in environ- ments where the individual run performs poorly (Humanoid and HumanoidStandup)
beneficial (Ant,
26

---

Published as a conference paper at ICLR 2025
D.6 ON-THE-FLY HYPERPARAMETER SELECTION ON ATARI
On-The-Fip Thy-FFyyy activation function selection on Atari
AdaDQN ReLU Tanh
Sigmoid
BattleZone
DoubleDunk
NameThisGane ThisGame
2.5
0.5
0.0
0.0
0.0
2.5
Target
1QM Human Norm Score
On-The- Fll architecture selection on Atari
AdaDQN
- 0 [32, 64, 64, 512] - [44, 52, 64, 512]
[46, 46, 46, 714]
BattleZone
DoubleDunk
Name NameThis ThisGame
2.5
0.5
0.0
0.0
Target
On-The-Fly Adam's epsilon selection on Atari
AdaDQN
-- EAdam: 1.5c-4 -. EAdom: 1.5c-6
EAdam: 1.5c-8
BattleZone
DoubleDunk
NameThisGame
Name ThisGame
0.5
0.5
0
0.0
Target
10
20
30
40
10
20
30
40
10
20
30
40
Num Frames (in millions)
Num Frames (in millions)
Num Frames (in millions)
Figure 20: AdaDQN performs similarly the best individual hyperparameters when selecting from 3 activation functions (left), architectures (middle), or 3 different values of Adam's 's epsilon param- eter (right). Below each performance plot, bar plot presents the distribution of the hyperparameters selected for computing the target.
AdaDQN
SEARL
DEHB Boxing
Random search
BattleZone
Enduro
Budget 40M
Budget 4IM
Budget 40M1
0.50
IQM Human Norm Score
0.25
0.00
Freeway
Pong
RoadRunner
Budger 40M
Budget 40M
Rudrett 40M
1.0
1.0
0.5
0.5
0.0
0.0
120
Num Frames millions)
Num Frumes millions)
Num Frames (in millions)
Figure 21: Per game performances AdaDQN and the baselines when the space of hyperparameters is infinite. We believe that the wide range of returns is due to the fact that the hyperparameter space large. Nevertheless, AdaDQN reaches higher IQM returns than SEARL on all games except on BattleZonee where only one seed performs well.
27

---

Published as a conference paper at ICLR 2025
BottleZone
Boxing
Enduro
High langet froq High HiP freq
High laruet lioq High HP freq
High targst tirqu
High HP Ereq
Low tarpet freq High Hir freq
0.
bag Ait won hog soaee yiill
Low targer freq Hilan lif frea
hag wory bag, plom
Low taged frea filae prrs
0,4
boy Mla--lt wart Seeg Semtinn 1ii
Low target froq
Low larget Ereq
Low larget Troq
Low HP freg
Low HP freq
Low HP freq
AdaDQN
SEARL max
SEARL- min
Freeway
Pong
RoadRunner
High turget freq
High target freq
High terget treq High HP freq
High HP freq
High HP fleq
Low ussest freq High HIP freq
L,02
beis NorT
bag jobm ypit
Low target free High Tithiiiee hiettee
0.9
hej lif work geu poaan yit
Low ungd freq Hiaai Sileen tons
I-6
wigh Cu aa New
lepou free
Low taE freq Low HP Boo
Law target freq Low HP freq
Low target free Low HP freq
Figure 22: Per game performances of AdaDQN and SEARL at 40M frames for different algorithm's 's hyperparameter values. As each online network is evaluated SEARL, we report the maximum and minimum observed return (SEARL- min and SEARL- max). Remarkably, AdaDQN's per- formances are always above the minimum performances of SEARL. In 4 out the considered games, AdaDQN outperforms SEARL maximum return all considered settings. High target freq corresponds to = 8000 and low target freq corresponds to T = 4000. HP freq is noted M in Figure 9, the high value is M 80000 and the low value is M 40000.
AdaDQN Pong
Target network
Online networks
Performances are low
Performances are increasing
MLPn layers CNN kemel size CNN n layers
CNN channels
27
Learning rate
Logcob
25
2.5
LOSS
23
21
19
0.0
CNN stridc
5
Stadardio
Trab
5831
Sofple:
Kihup
Sxhrux
4
Activation function
Opticttt
Novegad
Optimizer
2
3-
2
MLP I1 neurons
254
LegSolemo
Lodky
AMS3244
198
Sgrvee
GILU
AdX2no
142
CILI
Adiiicid
10
20
30
40
20
30
20
Num Frames (in millions)
Figure 23: Evolution of AdaDQN's hyperparameters hyyrraanneerrrenneeeteeeesr along the training for one seed on Pong. While
the online networks explore vast portion of the hyperparameter space, the target network focuses on a few values.
28


Reviews:
[{"review_id": "Ack16Ja4mC", "reviewer": "Reviewer_FXWZ", "summary": "The authors have developed a new AutoRL approach called Adaptive Q-Network (AdaQN), which aims to address the non-stationary optimization procedure in RL without requiring additional samples. The algorithm learns multiple Q-functions with different hyper-parameters while using the smallest approximation error as a shared target. So there's no need for additional samples. The paper also provides theoretical motivation that supports the algorithm and through experiments on Mujoco and Atari, AdaQN is shown to have better performance and sample efficiency.", "questions": "Can AdaQN effectively be applied on continuous action spaces (actor-critic frameworks) especially in complex high-dim environments? Which Q should be use to derive a policy network (maybe similar as we choose the one for target calculation)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simple but novel and effective design of training multiple Q functions with different hyper-parameters settings and target selection mechanism which means no additional samples are required for hyper-parameter tuning. \n2. AdaQN is shown to performance better in Mujoco and Atari benchmarks than DQN and other autoRL baselines\n3. Theoretical support is also presented to support the motivation of the selection mechanism in AdaQN.\n4. AdaQN can be applied to other critic-based algorithms", "weaknesses": "From my understanding, AdaQN's performance is still dependent on the initial range of hyper-parameters considered in the multiple Q setup. If this range is not representative of the optimal settings, AdaQN may not achieve the best possible performance. Also, it seems error or bias in the calculation of approximation error will affect the selection of the shared target.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cj17weUIJ6", "reviewer": "Reviewer_Kj2W", "summary": "This work proposes a new AutoRL algorithms that constructs an ensemble of Q-networks, each learned with different hyperaparameters (e.g., different learning rates). The main novelty is that during learning, there is an interaction between the different Q-networks; namely, the best performing Q-network is used for the target computation for all networks. This means that the individual performances of the networks will be different, potentially better than the performance obtained by just learning by scratch with a chosen hyperparameter. Moreover, the best performing hyperparameters may also change as learning progresses, highlighting an attractive property of the proposal. Action selection is also performed by picking the best performing Q-network from the previous transition, together with an $\\epsilon$ random selection of a Q-network from the ensemble. Experiments are performed on DQN lunar lander as well as SAC on MuJoCo tasks. They compare against other AutoRL algorithms like SEARL and DEHB, and obtain superior performance. The work also includes ablation studies on the components of the algorithm, e.g., the epsilon parameter, and the Q-network selection mechanism. Moreover, they provide other evidence, such as directly plotting which networks were selected. The final performances on the tasks seem reasonable, and the learning speed including hyperparameter selection is good compared to the baselines.", "questions": "Can you give more explanation on the evaluation protocol?\n\nWhat is the performance when running the optimal hyperparameters from the start?\n\nCan you clarify the parts that I listed in the weaknesses, including the discussion on the discount factor?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The main idea is clear, theoretically justified and seems like a good idea.\n- The performance improves significantly.\n- The experiments are thorough, and look at several metrics, include ablations, etc.\n- The exposition is good.", "weaknesses": "- The evaluation protocol was only briefly discussed, and was not completely clear to me. Particularly, how the baselines results are plotted could have been explained more prominently. I believe these plots include the hyperparameter selection cost on x-axis, but this could be clearer. Moreover, for reference, it would be good to know what the performance would be when running from scratch with the best performing hyperparameters.\n\nIn particular, what does this mean:\n> Additionally, we focus on sample efficiency; thus, for reporting the grid search results, we multiply the number of interactions of the best achieved performance by the number of individual trials, as advocated by Franke et al. (2021). We also report the average performance over the individual runs as the performance that a random search would obtain in expectation.\n\nThis bit seems very important for interpreting the results, but it's only briefly mentioned. I would suggest explaining this with equation notation to make it precise. Moreover, perhaps it would be useful to explain the hyperparameter tuning task setting, what the objective and metrics are in the background. Moreover, the axis labels such as \"Env steps\" can be confusing, because the same notation is used in standard RL where one just runs the best performing hyperparameters without considering cost of searching the hyperparameters. These points would be good to clarify.\n\n- Around the end of the paper, there was a part about also selecting the $\\gamma$ discount parameter, but this seemed unsound to me. The discount parameter affects the Q-function, so different Q-functions would necessarily require different targets, and I don’t think they could be shared. For example setting $\\gamma=0$ would probably give good small prediction errors, but would not actually perform well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "56jNKg8PtP", "reviewer": "Reviewer_PnAT", "summary": "This paper addresses the sensitivity of hyperparameters in deep Reinforcement Learning (RL) and proposes a method to speed up hyperparameter selection compared to previous AutoRL approaches. The authors introduce the Adaptive Q-Network (AdaQN), which learns several Q-functions, each trained with different hyperparameters. These Q-functions are updated online using the one with the smallest approximation error as a shared target. Comprehensive experiments on MuJoCo and Atari environments demonstrate the effectiveness of AdaQN.", "questions": "Given the motivation to \"speed up hyperparameters selection,\" how can the selected hyperparameters by AdaQN be used effectively? Can the selected hyperparameters by AdaQN be used to train a better DQN? It seems the improvements by AdaQN are due to the use of different Q-functions, similar to ensemble methods, with the difference being the use of various hyperparameters instead of just different initializations. Could the paper be motivated by performance improvement using different hyperparameters rather than speeding up hyperparameter selection?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses the crucial issue of hyperparameter selection in deep RL and provides an effective solution. The proposed method is well written and organized. Comprehensive experiments validate the effectiveness of the method.", "weaknesses": "Theorem 4.1 explains why the empirical Bellman operator in Eq (3) derived from experience replay is unbiased to the true Bellman operator in Eq (2). However, it lacks a theoretical analysis on why the selection scheme from Eq (2) leads to convergence rather than local minima. A formal theoretical analysis would strengthen the paper by explaining the convergence properties of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)