Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning

## Abstract

Abstract Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called Adaptive Q 𝑄 Q italic_Q -Network (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several Q 𝑄 Q italic_Q -functions, each one trained with different hyperparameters, which are updated online using the Q 𝑄 Q italic_Q -function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari 2600 2600 2600 2600 games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability.

## 1 Introduction

Deep Reinforcement Learning (RL) has proven effective at solving complex sequential decision problems in various domains (Mnih et al., ; Haarnoja et al., ; Silver et al., ). Despite their success in many fields, deep RL algorithms suffer from brittle behavior with respect to hyperparameters (Mahmood et al., ; Henderson et al., ). For this reason, the field of automated Reinforcement Learning (AutoRL) (Parker-Holder et al., ) has gained popularity in recent years. AutoRL methods aim to optimize the hyperparameter selection so that practitioners can avoid time-consuming hyperparameter searches. AutoRL methods also seek to minimize the number of required samples to achieve reasonable performances so that RL algorithms can be used in applications where a limited number of interactions with the environment is possible. AutoRL is still in its early stage of development, and most existing methods adapt techniques that have been shown to be effective for automated Machine Learning (AutoML) (Hutter et al., ; Falkner et al., ). However, RL brings additional challenges that have been overlooked till now, despite notoriously requiring special care (Igl et al., ). For example, due to the highly non-stationary nature of RL, it is unlikely that a static selection of hyperparameters works optimally for a given problem and algorithm (Mohan et al., ). This has led to the development of several techniques for adapting the training process online to prevent issues like local minima (Nikishin et al., ; Sokar et al., ), lack of exploration (Klink et al., ), and catastrophic forgetting (Kirkpatrick et al., ).

In this work, we introduce a novel approach for AutoRL to improve the effectiveness of learning algorithms by coping with the non-stationarities of the RL optimization procedure. Our investigation stems from the intuition that the effectiveness of each hyperparameter selection changes dynamically after each training update. Building upon this observation, we propose to adaptively select the best-performing hyperparameters configuration to carry out Bellman updates. To highlight this idea, we call our approach Adaptive $Q$-Network (AdaQN). In practice, AdaQN uses an ensemble of $Q$-functions, trained with different hyperparameters, and selects the one with the smallest approximation error (Schaul et al., ; D’Eramo & Chalvatzaki, ) to be used as a shared target for each Bellman update. By doing this, our method considers several sets of hyperparameters at once each time the stationarity of the optimization problem breaks, i.e., at each target update (Figure 1). In the following, we provide a theoretical motivation for our selection mechanism and then, we empirically validate AdaQN in MuJoCo control problems and Atari $2600$ games, showing that it has several advantages over carrying out individual runs with static hyperparameters. Importantly, by trading off different hyperparameters configurations dynamically, AdaQN not only can match the performance of the best hyperparameter selection at no cost of additional samples, but can also reach superior overall performance than any individual run. Finally, we compare AdaQN against strong AutoRL baselines and evince AdaQN’s superiority in finding good hyperparameters in a sample-efficient manner.

Figure: Figure 1: Left: Each line represents a training of $Q$-Network (QN) with different hyperparameters. Right: At the $i^{\text{th}}$ target update, Adaptive $Q$-Network (AdaQN) selects the network $Q_{i}$ (highlighted with a crown) that is the closest to the previous target $\Gamma\bar{Q}_{i-1}$, where $\Gamma$ is the Bellman operator.
Refer to caption: x1.png

## 2 Preliminaries

We consider discounted Markov decision processes (MDPs) defined as $\mathcal{M}=\langle\mathcal{S}$, $\mathcal{A}$, $\mathcal{P}$, $\mathcal{R}$, $\gamma\rangle$, where $\mathcal{S}$ and $\mathcal{A}$ are measurable state and action spaces, $\mathcal{P}:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$(^1^11$\Delta(\mathcal{X})$ denotes the set of probability measures over a set $\mathcal{X}$.) is a transition kernel, $\mathcal{R}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ is a reward function, and $\gamma\in[0,1)$ is a discount factor (Puterman, ). A policy is a function $\pi:\mathcal{S}\to\Delta(\mathcal{A})$, inducing an action-value function $Q^{\pi}(s,a)\triangleq\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}
\mathcal{R}(s_{t},a_{t})|s_{0}=s,a_{0}=a\right]$ that gives the expected discounted cumulative return executing action $a$ in state $s$, following policy $\pi$ thereafter. The objective is to find an optimal policy $\pi^{*}=\operatorname*{arg\,max}_{\pi}V^{\pi}(\>\cdot\>)$, where $V^{\pi}(\>\cdot\>)=\mathbb{E}_{a\sim\pi(\>\cdot\>)}[Q^{\pi}(\>\cdot\>,a)]$. Approximate value iteration (AVI) and approximate policy iteration (API) are two paradigms to tackle this problem (Sutton & Barto, ). While AVI aims at estimating the optimal action-value function $Q^{*}$, i.e., the action-value function of the optimal policy, API is a two-step procedure that alternates between Approximate policy evaluation (APE), a method to evaluate the action-value function $Q^{\pi}$ of the current policy $\pi$, and policy improvement, which updates the current policy by taking greedy actions on $Q^{\pi}$.

In this work, we focus on AVI and APE. Both aim to solve a Bellman equation, whose solution is $Q^{*}$ for AVI and $Q^{\pi}$ for APE. Those solutions are the fixed point of a Bellman operator $\Gamma$, where $\Gamma$ is the optimal Bellman operator $\Gamma^{*}$ for AVI, and the Bellman operator $\Gamma^{\pi}$ for APE. For a $Q$-function $Q$, a state $s$ and an action $a$, $\Gamma^{*}Q(s,a)=r+\gamma\mathbb{E}_{s^{\prime}\sim\mathcal{P}(s,a)}[\max_{a^{
\prime}}Q(s^{\prime},a^{\prime})]$ and $\Gamma^{\pi}Q(s,a)=r+\gamma\mathbb{E}_{s^{\prime}\sim\mathcal{P}(s,a),a^{
\prime}\sim\pi(s^{\prime})}[Q(s^{\prime},a^{\prime})]$. $\Gamma^{*}$ and $\Gamma^{\pi}$ are $\gamma$-contraction mapping in the infinite norm. Because of this, these two methods repeatedly apply their associated Bellman operator, starting from a random $Q$-function. The fixed point theorem ensures that each iteration is closer to the fixed point of the respective Bellman equation. Hence, the more Bellman iterations are performed, the closer the iterated $Q$-function will be to the desired action-value function. In model-free RL, $\Gamma^{*}$ and $\Gamma^{\pi}$ are approximated by their empirical version that we note $\hat{\Gamma}$ without distinguishing between AVI and APE since the nature of $\hat{\Gamma}$ can be understood from the context. We denote $\Theta$, the space of $Q$-functions parameters. In common $Q$-Network (QN) approaches, given a fixed vector of parameters $\bar{\theta}\in\Theta$, another vector of parameters $\theta$ is learned to minimize

$$ $\mathcal{L}_{\text{QN}}(\theta|\bar{\theta},s,a,r,s^{\prime})=(\hat{\Gamma}Q_{ \bar{\theta}}(s,a)-Q_{\theta}(s,a))^{2}$ (1) $$

for a sample $(s,a,r,s^{\prime})$. $Q_{\bar{\theta}}$ is usually called the target $Q$-function because it is used to compute the target $\hat{\Gamma}Q_{\bar{\theta}}$ while $Q_{\theta}$ is called the online $Q$-function. We refer to this step as the projection step since the Bellman iteration $\Gamma Q_{\bar{\theta}}$ is projected back into the space of $Q$-functions that can be represented by our function approximations. The target parameters $\bar{\theta}$ are regularly updated to the online parameters $\theta$ so that the following Bellman iterations are learned. This step is called the target update. We note $\bar{\theta}_{i}$ the target parameters after the $i^{\text{th}}$ target update.

## 3 Related work

Methods for AVI or APE (Mnih et al., ; Dabney et al., ; Haarnoja et al., ; Bhatt et al., ) are highly dependent on hyperparameters (Henderson et al., ; Andrychowicz et al., ; Engstrom et al., ). AutoRL tackles this limitation by searching for effective hyperparameters.

We follow the categorization of AutoRL methods presented in Parker-Holder et al. () to position our work in the AutoRL landscape. Contrary to AdaQN, many approaches consider optimizing the hyperparameters through multiple trials. A classic approach is to cover the search space with a grid search or a random search (Hutter et al., ; Bergstra & Bengio, ). Some other methods use Bayesian optimization to guide the search in the space of hyperparameters (Chen et al., ; Falkner et al., ; Nguyen et al., ; Shala et al., ), leveraging the information collected from individual trials. Those methods tackle the problem of finding the best set of hyperparameters but do not consider changing the hyperparameters during a single trial, which would be more appropriate for handling the non-stationarities inherent to the RL problem. To this end, evolutionary approaches have been developed (Stanley et al., ; Jaderberg et al., ; Awad et al., ), where the best elements of a population of agents undergo genetic modifications during training. As an example, in SEARL (Franke et al., ), a population of agents is first evaluated in the environment. Then, the best-performing agents are selected, and a mutation process is performed to form the next generation of agents. Finally, the samples collected during evaluation are taken from a shared replay buffer to train each individual agent, and the loop repeats. Even if selecting agents based on their performance in the environment seems reasonable, we argue that it hinders sample-efficiency since exploration in the space of hyperparameters might lead to poorly performing agents. Moreover, if poorly performing agents are evaluated, low-quality samples will then be stored in the replay buffer and used later for training, which could lead to long-term negative effects. This is why we propose to base the selection mechanism on the approximation error, which does not require additional interaction with the environment. Other methods have considered evaluating the $Q$-functions offline (Tang & Wiens, ). Most approaches consider the Bellman error $\|\Gamma Q_{i}-Q_{i}\|$ instead of the approximation error $\|\Gamma Q_{i-1}-Q_{i}\|$, where $Q_{i}$ is the $Q$-function obtained after $i$ Bellman iterations (Farahmand & Szepesvári, ; Zitovsky et al., ). However, this approach considers the empirical estimate of the Bellman error which is a biased estimate of the true Bellman error (Baird, ). Lee et al. () propose to rely on the approximation error to choose between two different classes of functions in an algorithm called ModBE. AdaQN differs from ModBE since ModBE does not adapt the hyperparameters at each Bellman iteration but instead performs several training steps before comparing two classes of functions. Furthermore, the use of the approximation error can be theoretically justified by leveraging Theorem $3.4$ from Farahmand (), as shown in Section 4.

Meta-gradient reinforcement learning (MGRL) also optimizes the hyperparameters during training (Finn et al., ; Zahavy et al., ; Flennerhag et al., ). However, MGRL differs from our approach because MGRL focuses on tuning the target to ease the optimization process, while AdaQN uses diverse online networks to overcome optimization hurdles. This shift in paradigm leads AdaQN to avoid some of MGRL’s shortcomings. Indeed, AdaQN can work with discrete hyperparameters (e.g., optimizer, activation function, losses, or neural architecture), only requires first-order approximation, and does not suffer from non-stationarities when hyperparameters are updated since each online network is trained with respect to its own hyperparameters (Xu et al., ). Finally, AdaQN fits best in the ”Blackbox Online Tuning” cluster even if most methods in this cluster focus on the behavioral policy (Schaul et al., ; Badia et al., ) or build the hyperparameters as functions of the state of the environment (Sutton & Singh, ; White & White, ). For example, Riquelme et al. () develop a method called adaptive TD that selects between the TD-update or the Monte-Carlo update depending on whether the TD-update belongs to a confidence interval computed from several Monte-Carlo estimates.

## 4 Adaptive temporal-difference target selection

Notoriously, when the AVI/APE projection step described in Section 2 is carried out for non-linear function approximation together with off-policy sampling, convergence to the fixed point is no longer guaranteed. Nevertheless, well-established theoretical results in AVI relate the approximation error $\|\Gamma Q_{\bar{\theta}_{i-1}}-Q_{\bar{\theta}_{i}}\|$ at each target update $i$ to the performance loss $\|Q^{*}-Q^{\pi_{N}}\|$, i.e., the distance between the $Q$-function at timestep $N$ during the training and the optimal one $Q^{*}$. For a fixed number of Bellman iterations $N$, Theorem $3.4$ from Farahmand (), stated in Appendix A, demonstrates that the performance loss is upper bounded by a term that decreases when the sum of approximation errors $\sum_{i=1}^{N}\|\Gamma Q_{\bar{\theta}_{i-1}}-Q_{\bar{\theta}_{i}}\|$ decreases, i.e., every projection step improves. Hence, minimizing the sum of approximation errors is crucial to get closer to the fixed point. However, due to the complexity of the landscape of the empirical loss $\mathcal{L}_{\text{QN}}$ (Mohan et al., ), using fixed hyperparameters to minimize each approximation error $\|\Gamma Q_{\bar{\theta}_{i-1}}-Q_{\bar{\theta}_{i}}\|$ is unlikely to be effective and can, in the worst case, lead to local minima (Nikishin et al., ; Sokar et al., ) or diverging behaviors (Baird, ).

In this work, we tackle this problem by introducing a method that can seamlessly handle multiple hyperparameters in a single AVI/APE training procedure to speed up the reduction of the performance loss described above, compared to a fixed hyperparameter selection. This method, which we call Adaptive $Q$-Network (AdaQN), learns several online networks trained with different hyperparameters. Crucially, to cope with the non-stationarity of the RL procedure, at each target update, the online network with the lowest approximation error and thus the lowest performance loss, is selected as a shared target network used to train all the online networks. This results in moving all the networks of the ensemble towards the one that is currently the closest to the fixed point, thus benefiting from the heterogeneous ensemble to speed up learning.

We denote $(\theta^{k})_{k=1}^{K}$ the parameters of the online networks and $\bar{\theta}_{i}$ the parameters of the shared target network after the $i^{\text{th}}$ target update. To minimize the $i^{\text{th}}$ approximation error, the optimal choice for $\bar{\theta}_{i}$ given $\bar{\theta}_{i-1}$, is the index of the closest online network from the target $\Gamma Q_{\bar{\theta}_{i-1}}$:

$$ $\bar{\theta}_{i}\leftarrow\theta^{\psi}\text{ where }\psi=\operatorname*{arg\, min}_{k\in\{1,\ldots,K\}}\|\Gamma Q_{\bar{\theta}_{i-1}}-Q_{\theta^{k}}\|_{2, \nu}^{2},$ (2) $$

where $\nu$ is the distribution of state-action pairs in the replay buffer. Note that Equation (2) contains the true Bellman operator $\Gamma$ which is not known. Instead, we propose to rely on the value of the empirical loss to select the next target network:

$$ $\bar{\theta}_{i}\leftarrow\theta^{\psi}\text{ where }\psi=\operatorname*{arg\, min}_{k\in\{1,\ldots,K\}}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\mathcal{L}_{ \text{QN}}(\theta^{k}|\bar{\theta}_{i-1},s,a,r,s^{\prime}),$ (3) $$

where $\mathcal{D}$ is the replay buffer. Theorem 4.1 shows under which condition the selected index in Equation (3) is the same as the one in Equation (2). Importantly, this condition is valid when the dataset is infinite and the samples are generated from the true distribution. Indeed, the empirical Bellman operator is unbiased w.r.t. the true Bellman operator. The proof, inspired by the bias-variance trade-off in supervised learning, can be found in Appendix A.

###### Theorem 4.1 .

Let $(\theta^{k})_{k=1}^{K}\in\Theta^{K}$ and $\bar{\theta}\in\Theta$ be vectors of parameters representing $K+1$ $Q$-functions. Let $\mathcal{D}=\{(s,a,r,s^{\prime})\}$ be a set of samples. Let $\nu$ be the distribution represented by the state-action pairs present in $\mathcal{D}$. We note $\mathcal{D}_{s,a}=\{(r,s^{\prime})|(s,a,r,s^{\prime})\in\mathcal{D}\},\forall(
s,a)\in\mathcal{D}$.
If for every state-action pair $(s,a)\in\mathcal{D}$, $\mathbb{E}_{(r,s^{\prime})\sim\mathcal{D}_{s,a}}\left[\hat{\Gamma}Q_{\bar{
\theta}}(s,a)\right]=\Gamma Q_{\bar{\theta}}(s,a)$, then we have

$$ $\operatorname*{arg\,min}_{k\in\{1,\ldots,K\}}||\Gamma Q_{\bar{\theta}}-Q_{ \theta^{k}}||_{2,\nu}^{2}=\operatorname*{arg\,min}_{k\in\{1,\ldots,K\}}\sum_{( s,a,r,s^{\prime})\in\mathcal{D}}\mathcal{L}_{\text{QN}}(\theta^{k}|\bar{\theta },s,a,r,s^{\prime}).$ $$

Key to our approach is that each vector of parameters $\theta^{k}$ can be trained with a different optimizer, learning rate, architecture, activation function, or any other hyperparameter that only affects its training. By cleverly selecting the next target network from a set of diverse online networks, AdaQN has a higher chance of overcoming the typical challenges of optimization presented earlier.

###### Theorem 4.1 .

### 4.1 Algorithmic implementation

Multiple algorithms can be derived from our formulation. Algorithm 1 shows an adaptive version of Deep $Q$-Network (DQN, Mnih et al. ()) that we call Adaptive Deep $Q$-Network (AdaDQN). Similarly, Adaptive Soft Actor-Critic (AdaSAC), presented in Algorithm 2, is an adaptive version of Soft Actor-Critic (SAC, Haarnoja et al. ()). In SAC, the target network is updated using Polyak averaging (Lillicrap et al., ). Therefore, we consider $K$ target networks $(\bar{\theta}^{k})_{k=1}^{K}$. Each target network $\bar{\theta}^{k}$ is updated from its respective online network $\theta^{k}$, as shown in Line 11 of Algorithm 2. However, each online network is trained w.r.t. a shared target chosen from a single target that comes from the set of $K$ target networks. Similarly to the strategy presented in Equation (3), after the $i^{\text{th}}$ target update, the shared target network is chosen as

$$ $\bar{\theta}_{i}\leftarrow\bar{\theta}^{\psi}\text{ where }\psi=\operatorname* {arg\,min}_{k\in\{1,\ldots,K\}}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\mathcal {L}_{\text{QN}}(\theta^{k}|\bar{\theta}_{i-1},s,a,r,s^{\prime}).$ (4) $$

Figure: Algorithm 1 Adaptive Deep $Q$-Network (AdaDQN). Modifications to DQN are marked in purple.

Multiple online networks enable to choose which network to use to sample actions in a value-based setting (see Line 3 of Algorithm 1) and which network is selected to train the actor in an actor-critic setting (see Line 13 of Algorithm 2). This choice is related to the behavioral policy, thus, we choose the same strategy in both settings. Intuitively, the optimal choice would be to pick the best-performing network; however, this information is not available. More importantly, only exploring using the best-performing network could lead the other online networks to learn passively (Section 5.1). This would make the performances of the other networks drop (Ostrovski et al., ), making them useless for the rest of the training. Inspired by $\epsilon$-greedy policies commonly used in RL for exploration, we select a random network with probability $\epsilon_{b}$ and select the online network corresponding to the selected target network (as a proxy for the best-performing network) with probability $1-\epsilon_{b}$. We use a linear decaying schedule for $\epsilon_{b}$. Finally, at each target update, the running loss accumulated after each gradient step is used instead of the loss computed over the entire dataset.

## 5 Experiments

We first evaluate our method when hyperparameters are selected from a finite space. This setting is relevant when the RL practitioner has a good understanding of most hyperparameters and only hesitates among a few ones. We compare AdaQN to an exhaustive grid search where both approaches have access to the same sets of hyperparameters. This allows us to see how AdaQN performs compared to the best static set of hyperparameters. Additionally, we focus on sample efficiency; thus, for reporting the grid search results, we multiply the number of interactions of the best achieved performance by the number of individual trials, as advocated by Franke et al. (). We also report the average performance over the individual runs as the performance that a random search would obtain in expectation. Then, we evaluate our method in a second setting for which the hyperparameter space is infinite. This setting corresponds to the more general case where the RL practitioner has to select hyperparameters for a new environment, having limited intuitions for most hyperparameters. We compare AdaQN to SEARL (Franke et al. () presented in Section 3), which is specifically designed for this setting, DEHB (Awad et al., ), which is a black box optimizer that has been shown to be effective for RL (Eimer et al., ), and random search, which remains a very challenging baseline for hyperparameter optimization. In both settings, the remaining hyperparameters are kept unchanged. Appendix B describes all the hyperparameters used for the experiments in Table 2, 2 and 4 along with the description of the environment setups. The code is based on the Stable Baselines implementation (Raffin et al., ) and Dopamine RL (Castro et al., )(^2^22The code is available in the supplementary material and will be made open source upon acceptance.).

Performance metric. As recommended by Agarwal et al. (), we show the interquartile mean (IQM) along with shaded regions showing pointwise $95\%$ percentile stratified bootstrap confidence intervals. IQM trades off the mean and the median where the tail of the score distribution is removed on both sides to consider only $50\%$ of the runs. We argue that the final score is not enough to properly compare RL algorithms since methods that show higher initial performances are better suited for real-world experiments compared to methods that only show higher performances later during training. This is why we also analyze the performances with the Area Under the Curve (AUC) that computes the integral of the IQM along the training. We also report the worst-performing seed to analyze the robustness of AdaQN w.r.t. stochasticity and highlight AdaQN’s stability against hyperparameter changes. We use $20$ seeds for Lunar Lander (Brockman et al., ), $9$ seeds for MuJoCo (Todorov et al., ), and $5$ seeds for Atari (Bellemare et al., ).

### 5.1 A proof of concept

Figure: Figure 2: On-the-fly architecture selection on Lunar Lander. All architectures contain two hidden layers. The number of neurons in each layer is indicated in the legend. Left: AdaDQN yields a better AUC than every DQN run. Right: Ablation on the behavioral policy and on the strategy to select the target network used to compute the target. Each version of AdaDQN uses the $4$ presented architectures. The strategy presented in Equation (3) outperforms the other considered strategies.
Refer to caption: x2.png

Figure: Figure 3: Distribution of the selected hyperparameters for the target network (top) and for the behavioral policy (bottom) across all seeds. Left: AdaDQN mainly selects the hyperparameter that performs best when evaluated individually. Middle: AdaDQN with $\epsilon_{b}=0$ also focuses on the best individual architecture but does not use all available networks for sampling actions, which lowers its performance. Right: AdaDQN-max is a version of AdaDQN where the minimum operator is replaced by the maximum operator for selecting the following target network.
Refer to caption: x3.png

We consider $4$ different architectures for AdaDQN and compare their performances to the individual runs in Figure 2 (left). The $4$ architectures are composed of two hidden layers, with the number of neurons indicated in the legend. Interestingly, AdaDQN outperforms the best individual architecture, meaning that by selecting different architectures during the training, AdaDQN better copes with the non-stationarities of the optimization procedure. Figure 2 (right) shows the performances for AdaDQN along with a version of AdaQN where $\epsilon_{b}=0$ during the training (AdaDQN $\epsilon_{b}=0$), i.e. actions are always sampled for the online network that was chosen for computing the target. This variant of AdaQN performs similarly to the best individual run but underperforms compared to AdaDQN because it suffers from passive learning as explained in Section 4.1. Additionally, we evaluate a version where the target network is selected randomly from the set of online networks, calling this variant RandDQN. This version is similar to the way the target is computed in RED$Q$ (Chen et al., ). While sampling uniformly from similar agents yields better performance, as shown in Chen et al. (), this strategy suffers when one agent is not performing well. In our case, DQN $[25,25]$ harms the overall performance. Finally, taking the maximum instead of the minimum to select the target (AdaDQN-max) performs as badly as the worst available agent (i.e., DQN $[25,25]$).

In Figure 3 (top), we analyze the distribution of the selected hyperparameters for the target network across all seeds along the training. Importantly, AdaDQN does not always select the architecture that performs the best when trained individually, as the network DQN $[100,100]$ is sometimes chosen instead of DQN $[200,200]$. On the right, AdaDQN-max mainly selects DQN $[25,25]$, which explains its poor performance. The gap in performance between AdaDQN and AdaDQN with $\epsilon_{b}=0$ can be understood by the bottom row of the figure. It shows the distribution of the selected hyperparameters for sampling the actions. Setting $\epsilon_{b}=0$ forces the networks not selected for computing the target to learn passively (the top and bottom bar plots are the same for AdaDQN with $\epsilon_{b}=0$). On the contrary, AdaDQN benefits from a better exploration at the beginning of the training (the bottom bar plot is more diverse than the top one at the beginning of the training).

### 5.2 Continuous control: MuJoCo environments

Figure: Figure 4: On-the-fly hyperparameter selection on MuJoCo. The $16$ sets of hyperparameters are the elements of the Cartesian product between the learning rates $\{0.0005,0.001\}$, the optimizers $\{$Adam, RMSProp$\}$, the critic’s architectures $\{[256,256],[512,512]\}$ and the activation functions $\{$ReLU, Sigmoid$\}$. Left: AdaSAC is more sample-efficient than random search and grid search. Right: AdaSAC yields a better AUC than every SAC run while having a greater final score than $13$ out of $16$ SAC runs. The shading of the dashed lines indicates their ranking for the AUC metric.
Refer to caption: x4.png

We evaluate AdaSAC for a wider choice of hyperparameters on $6$ MuJoCo environments. We consider selecting from $16$ sets of hyperparameters. Those sets form the Cartesian product between the learning rates $\{0.0005,0.001\}$, the optimizers $\{$Adam, RMSProp$\}$, the critic’s architectures $\{[256,256],[512,512]\}$ and the activation functions $\{$ReLU, Sigmoid$\}$. This sets $K$ to $2^{4}=16$. The values of the hyperparameters were chosen to be representative of common choices made when tuning by hand a SAC agent. In Figure 4 (left), we compare AdaSAC with a grid search performed on the $16$ set of hyperparameters. AdaSAC is an order of magnitude more sample efficient than grid search and outperforms random search all along the training. Notably, AdaSAC’s worst-performing seed performs on par with the random search approach while greatly outperforming the worst-performing seed of the grid search. In Figure 4 (right), we show AdaSAC’s performance along with the individual performance of each set of hyperparameters. AdaSAC yields the highest AUC metric while outperforming $13$ out of the $16$ individual runs in terms of final performance. Interestingly, AdaSAC reaches the performance of vanilla SAC in less than half of the samples. We also show the performance of RandSAC, for which a random target network is selected to compute the target instead of following the strategy presented in Equation (4). For clarity, the labels of the $16$ individual runs are not shown. They are available in Figure 11 of Appendix D.1.

Figure: Figure 5: Per environment IQM return when AdaSAC and RandSAC select from the $16$ sets of hyperparameters described in Figure 4. Below each performance plot, a bar plot presents the distribution of hyperparameters selected for the target network across all seeds. AdaSAC outperforms RandSAC and most individual run by designing non-trivial hyperparameter schedules.
Refer to caption: x5.png

We now analyze which set of hyperparameters AdaSAC selects to compute the target. For each environment, we show in Figure 5 the performances of each hyperparameter when trained individually, AdaSAC and RandSAC performances. Below each performance plot, we report the distribution of hyperparameters selected to compute the target across all seeds. Overall, the selected networks are changing during the training, which indicates that the loss is not always minimized by the same set of hyperparameters. This supports the idea of selecting the target network based on Equation (3). Furthermore, the selected networks are not the same across the different environments. This shows that AdaSAC designs non-trivial hyperparameter schedules that cannot be handcrafted. On Humanoid, selecting the network corresponding to the best-performing agent leads to similar performances to the best-performing agents when trained individually. On HumanoidStandup, the selection strategy avoids selecting the worst-performing agents. This is why AdaSAC outperforms RandSAC, which blindly selects agents. HalfCheetah is the only environment where RandSAC slightly outperforms AdaSAC. AdaSAC still yields a better final performance than $6$ of the individual runs. Finally, on Hopper and Walker2d, AdaSAC outperforms every individual run for the AUC metric by mainly selecting hyperparameters that are not performing well when trained individually. The ability of AdaSAC to adapt the hyperparameters at each target update enables it to better fit the targets, which yields better performances.

Ablations. We evaluate AdaSAC against random search and grid search in $4$ different scenarios where only one hyperparameter changes. The results presented in Appendices D.2, D.3, D.4, and D.5 show that AdaSAC is robust to variations of the hyperparameters. We first give $4$ architectures of different capacities as input. Then, we run AdaSAC with $3$ different learning rates, and we also run another study where AdaSAC is given $3$ different optimizers. In each case, AdaSAC matches the best-performing individual runs while successfully ignoring the worst-performing sets of hyperparameters. Finally, we evaluate AdaSAC with $3$ different activation functions given as input. This time, AdaSAC outperforms $2$ out of the $3$ individual runs. Interestingly, RandSAC seems to suffer more in that setting by only outperforming the worst-performing individual run. This further demonstrates that the strategy presented in Equation (3) is effective.

### 5.3 Vision-based control: Atari 2600 2600 2600 2600

Figure: Figure 6: On-the-fly hyperparameter selection on $3$ Atari games. AdaDQN performs similarly to the best individual hyperparameters when selecting from $3$ activation functions (left), $3$ architectures (middle), or $3$ different values of Adam’s epsilon parameter (right). Below each performance plot, a bar plot presents the distribution of the hyperparameters selected for computing the target.
Refer to caption: x6.png

We provide an additional experiment on $3$ games of the Atari benchmark. Similarly to the ablation study made on the MuJoCo benchmark, we conclude, in light of Figure 6, that AdaDQN performs on par with the best-performing hyperparameter when the space of hyperparameters is composed of different activation functions (left), architectures (middle) or different values of Adam’s epsilon parameter (right). This means that random search and grid search underperform compared to AdaDQN. Importantly, the common hyperparameters used for DQN (in orange) are not the best-performing ones in all settings, which motivates using Auto RL. Below each performance plot, we report the distribution of the hyperparameters selected for computing the target. Remarkably, AdaDQN never selects Sigmoid activations, which perform poorly when evaluated individually. Interestingly, for the ablation study on the architecture and Adam’s epsilon parameter, AdaDQN uses all available hyperparameters when they perform equally well. We note that, despite not being often tuned, Adam’s epsilon parameter plays a role in yielding high returns as observed in Obando-Ceron et al. (). Figure 20 reports the decomposition of Figure 6 into the $3$ considered Atari games.

### 5.4 Infinite hyperparameter spaces

Figure: Figure 7: On-the-fly hyperparameter selection on $6$ Atari games. Each method has to find well-performing hyperparameters in a very challenging hyperparameter space composed of $18$ activation functions, $24$ optimizers, $4$ losses, a wide range of learning rates, and fully parameterizable architectures. Left: AdaDQN outperforms SEARL, random search, and grid search in the limited budget of $40$M frames and keeps increasing its performance afterward. Right: AdaDQN is robust to hyperparameter changes. Each axis of the radar plot shows the performances at $40$M frames.

We now focus on a setting in which the space of hyperparameters is infinite. We design a wide space of hyperparameters to come closer to the goal of removing the need for prior knowledge about the algorithm and environment. The hyperparameter space, presented in Table 4, comprises $18$ activation functions, $24$ optimizers, $4$ losses, a wide range of learning rates, and fully parameterizable architectures. We let SEARL and AdaDQN train $5$ neural networks in parallel. Figure 9 shows how AdaDQN and SEARL differ. To generalize AdaQN to infinite hyperparameter spaces, we replace the RL training step in SEARL’s pipeline with the one described in Section 4, and the approximation errors are used as fitness values instead of the online evaluation of each network. The exploitation and exploration steps remain identical. We fix a budget of $40$M frames on $6$ games of the Atari benchmark. We reduce the budget to $30$M frames for random search to favor more individual trials, $6$ in total, as the hyperparameter space is large. Finally, we let DEHB benefit from its multi-fidelity component by allowing a varying budget from $20$M to $40$M frames. We report the maximum return observed since the beginning of the training for each seed of random search and DEHB. As every network is evaluated in SEARL, we report the maximum and minimum return observed at each evaluation step (SEARL - max and SEARL - min).

In Figure 7 (left), at $40$M frames, AdaDQN outperforms SEARL and matches the end performance of random search and DEHB. When trained for longer, AdaDQN continues to increase its performance and outperforms random search and DEHB before reaching $100$M frames. We argue that SEARL’s low performance is largely explained by the fact that many environment iterations are wasted by poorly performing networks (see SEARL - min), which also affects the other networks as the replay buffer is shared among them. Instead, AdaDQN only uses the networks that have a low training loss. Random search and DEHB usually require many environment interactions to perform well. We believe that having a wide hyperparameter space exacerbates this phenomenon. Figure 7 (right) highlights AdaDQN’s robustness w.r.t. its hyperparameters. Notably, AdaDQN reaches a higher IQM return than SEARL in the $4$ considered settings at $40$M frames. Figure 21 and 22 show the individual game performances. Figure 23 reports the evolution of the hyperparameters during AdaDQN’s training for one seed on Pong. Interestingly, the performances increase (green background) when the network trained on the Huber loss with ReLU activations and Adam optimizer is selected as target network.

Figure: Figure 8: On-the-fly discount factor selection on $2$ Atari games. AdaDQN and Meta-gradient DQN perform similarly. On the right of each performance plot, we report the evolution of the discount factor selected by AdaDQN and Meta-gradient DQN.
Refer to caption: x8.png

AdaQN can also be used to select the target hyperparameters. We propose to select from different discount factors sampled in $[0.985,0.995]$. We consider $K=5$ online networks and compare AdaDQN against Meta-gradient reinforcement learning (MGRL, Xu et al. ()). The cross-validated discount factor of MGRL is set at $0.99$, and the same discount factor is considered when selecting the next target network in AdaQN. Figure 8 shows that the two approaches reach similar performances on $2$ Atari games. Nevertheless, we recall that AdaQN differs from MGRL in many points (see Section 3). Crucially, it is not limited to continuous and differentiable hyperparameters.

## 6 Discussion and conclusion

We have introduced Adaptive $Q$-Network (AdaQN), a new method that trains an ensemble of diverse $Q$-networks w.r.t. a shared target, selected as the target network corresponding to the most accurate online network. We demonstrated that AdaQN is theoretically sound and we devised its algorithmic implementation. Our experiments show that AdaQN outperforms competitive AutoRL baselines in terms of sample efficiency, final performance, and robustness.
Limitations. Our work focuses on the agent’s hyperparameters; thus, environment’s properties, e.g., reward, cannot be handled by AdaQN. Nevertheless, an off-the-shelf AutoRL algorithm can be combined with AdaQN to handle these properties. Moreover, since AdaQN considers multiple $Q$-functions in the loss, the training time and memory requirements increase. We provide an analysis in Appendix C to show that this increase remains reasonable compared to the gain in performance and sample-efficiency. We stress that, in theory, the computations can be parallelized so that the adaptive version of an algorithm requires the same amount of time as its original algorithm. The additional ”for loop” in Line 9 of Algorithm 1 and the one in Line 7 of Algorithm 2 can be run in parallel as long as enough parallel processing capacity is available, as it is common in modern GPUs. Finally, instead of selecting one $Q$-function for computing the target, future works could consider a mixture of $Q$-functions learned with different hyperparameters as presented in Seyde et al. ().

## Appendix A Theorem statements and proofs

###### Theorem 3.4 from Farahmand ( 2011 ) .

Let $N\in\mathbb{N}^{*}$, and $\rho$, $\nu$ two probability measures on $\mathcal{S}\times\mathcal{A}$. For any sequence $(\bar{\theta}_{i})_{i=0}^{N}\in\Theta^{N+1}$ where $R_{\gamma}$ depends on the reward function and the discount factor, we have

$$ $\|Q^{*}-Q^{\pi_{N}}\|_{1,\rho}\leq C_{N,\gamma,R_{\gamma}}+\inf_{r\in[0,1]}F(r ;N,\rho,\gamma)\bigg{(}\textstyle\sum_{i=1}^{N}\alpha_{i}^{2r}\|\Gamma^{*}Q_{ \bar{\theta}_{i-1}}-Q_{\bar{\theta}_{i}}\|_{2,\nu}^{2}\bigg{)}^{\frac{1}{2}}$ $$

where $C_{N,\gamma,R_{\gamma}}$, $F(r;N,\rho,\gamma)$, and $(\alpha_{i})_{i=0}^{N}$ do not depend on $(\bar{\theta}_{i})_{i=0}^{N}$. $\pi_{N}$ is a greedy policy computed from $Q_{\theta_{N}}$.

###### Theorem 4.1 .

Let $(\theta^{k})_{k=1}^{K}\in\Theta^{K}$ and $\bar{\theta}\in\Theta$ be vectors of parameters representing $K+1$ $Q$-functions. Let $\mathcal{D}=\{(s,a,r,s^{\prime})\}$ be a set of samples. Let $\nu$ be the distribution represented by the state-action pairs present in $\mathcal{D}$. We note $\mathcal{D}_{s,a}=\{(r,s^{\prime})|(s,a,r,s^{\prime})\in\mathcal{D}\},\forall(
s,a)\in\mathcal{D}$.
If for every state-action pair $(s,a)\in\mathcal{D}$, $\mathbb{E}_{(r,s^{\prime})\sim\mathcal{D}_{s,a}}\left[\hat{\Gamma}Q_{\bar{
\theta}}(s,a)\right]=\Gamma Q_{\bar{\theta}}(s,a)$, then we have

$$ $\displaystyle\operatorname*{arg\,min}_{k\in\{1,\ldots,K\}}||\Gamma Q_{\bar{ \theta}}-Q_{\theta^{k}}||_{2,\nu}^{2}=\operatorname*{arg\,min}_{k\in\{1,\ldots ,K\}}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\mathcal{L}_{\text{QN}}(\theta^{k} |\bar{\theta},s,a,r,s^{\prime}).$ $$

###### Proof.

Let $(\theta^{k})_{k=1}^{K}\in\Theta^{K}$ and $\bar{\theta}\in\Theta$ be vectors of parameters representing $K+1$ $Q$-functions. Let $\mathcal{D}=\{(s,a,r,s^{\prime})\}$ be a set of samples. Let $\nu$ be the distribution of the state-action pairs present in $\mathcal{D}$. In this proof, we note $\hat{\Gamma}$ as $\hat{\Gamma}_{r,s^{\prime}}$ to stress its dependency on the reward $r$ and the next state $s^{\prime}$. For every state-action pair $(s,a)$ in $\mathcal{D}$, we define the set $\mathcal{D}_{s,a}=\{(r,s^{\prime})|(s,a,r,s^{\prime})\in\mathcal{D}\}$ and assume that $\mathbb{E}_{(r,s^{\prime})\sim\mathcal{D}_{s,a}}[\hat{\Gamma}_{r,s^{\prime}}Q_
{\bar{\theta}}(s,a)]=\Gamma Q_{\bar{\theta}}(s,a)$. Additionally, we note $M$ the cardinality of $\mathcal{D}$, $M_{s,a}$ the cardinality of $\mathcal{D}_{s,a}$ and $\mathring{\mathcal{D}}$ the set of unique state-action pairs in $\mathcal{D}$. We take $k\in\{1,\ldots,K\}$ and write

$$ $\displaystyle\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\mathcal{L}_{\text{QN}}( \theta^{k}|\bar{\theta},s,a,r,s^{\prime})$ $\displaystyle=\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left(\hat{\Gamma}_{r,s^{ \prime}}Q_{\bar{\theta}}(s,a)-Q_{\theta^{k}}(s,a)\right)^{2}$ $\displaystyle=\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left(\hat{\Gamma}_{r,s^{ \prime}}Q_{\bar{\theta}}(s,a)-\Gamma Q_{\bar{\theta}}(s,a)+\Gamma Q_{\bar{ \theta}}(s,a)-Q_{\theta^{k}}(s,a)\right)^{2}$ $\displaystyle={\color[rgb]{0,0,0}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left( \hat{\Gamma}_{r,s^{\prime}}Q_{\bar{\theta}}(s,a)-\Gamma Q_{\bar{\theta}}(s,a) \right)^{2}}$ $\displaystyle+$ $\displaystyle{\color[rgb]{0,0,0}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left( \Gamma Q_{\bar{\theta}}(s,a)-Q_{\bar{\theta}^{k}}(s,a)\right)^{2}}$ $\displaystyle+2$ $\displaystyle{\color[rgb]{0,0,0}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left( \hat{\Gamma}_{r,s^{\prime}}Q_{\bar{\theta}}(s,a)-\Gamma Q_{\bar{\theta}}(s,a) \right)\left(\Gamma Q_{\bar{\theta}}(s,a)-Q_{\theta^{k}}(s,a)\right)}.$ $$

The second last equation is obtained by introducing the term $\Gamma Q_{\bar{\theta}}(s,a)$ and removing it. The last equation is obtained by developing the previous squared term. Now, we study each of the three terms:

- •
${\color[rgb]{0,0,0}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left(\hat{\Gamma}_{
r,s^{\prime}}Q_{\bar{\theta}}(s,a)-\Gamma Q_{\bar{\theta}}(s,a)\right)^{2}}$ is independent of $\theta^{k}$
- •
${\color[rgb]{0,0,0}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left(\Gamma Q_{\bar
{\theta}}(s,a)-Q_{\theta^{k}}(s,a)\right)^{2}}$ equal to $M\times||\Gamma Q_{\bar{\theta}}-Q_{\theta^{k}}||_{2,\nu}^{2}$ by definition of $\nu$.
- •
$\displaystyle{\color[rgb]{0,0,0}\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\left(
\hat{\Gamma}_{r,s^{\prime}}Q_{\bar{\theta}}(s,a)-\Gamma Q_{\bar{\theta}}(s,a)
\right)\left(\Gamma Q_{\bar{\theta}}(s,a)-Q_{\theta^{k}}(s,a)\right)}$
$\displaystyle=\sum_{(s,a)\in\mathring{\mathcal{D}}}\left[\sum_{(r,s^{\prime})
\in\mathcal{D}_{s,a}}\left(\hat{\Gamma}_{r,s^{\prime}}Q_{\bar{\theta}}(s,a)-
\Gamma Q_{\bar{\theta}}(s,a)\right)\left(\Gamma Q_{\bar{\theta}}(s,a)-Q_{
\theta^{k}}(s,a)\right)\right]=0$
since, for every $(s,a)\in\mathring{\mathcal{D}}$,
$\displaystyle\sum_{(r,s^{\prime})\in\mathcal{D}_{s,a}}\left(\hat{\Gamma}_{r,s^
{\prime}}Q_{\bar{\theta}}(s,a)-\Gamma Q_{\bar{\theta}}(s,a)\right)$
$\displaystyle=M_{s,a}\left(\mathbb{E}_{(r,s^{\prime})\sim\mathcal{D}_{s,a}}[
\hat{\Gamma}_{r,s^{\prime}}Q_{\bar{\theta}}(s,a)]-\Gamma Q_{\bar{\theta}}(s,a)
\right)=0,$
the last equality holds from the assumption.

Thus, we have

$$ $\sum_{(s,a,r,s^{\prime})\in\mathcal{D}}\mathcal{L}_{\text{QN}}(\theta^{k}|\bar {\theta},s,a,r,s^{\prime})=\text{constant w.r.t }\theta^{k}+M\times||\Gamma Q_ {\bar{\theta}}-Q_{\theta^{k}}||_{2,\nu}^{2}$ $$

This is why

$$ $\operatorname*{arg\,min}_{k\in\{1,\ldots,K\}}\sum_{(s,a,r,s^{\prime})\in \mathcal{D}}\mathcal{L}_{\text{QN}}(\theta^{k}|\bar{\theta},s,a,r,s^{\prime})= \operatorname*{arg\,min}_{k\in\{1,\ldots,K\}}||\Gamma Q_{\bar{\theta}}-Q_{ \theta^{k}}||_{2,\nu}^{2}.$ $$

∎

###### Theorem 3.4 from Farahmand ( 2011 ) .

###### Theorem 4.1 .

###### Proof.

## Appendix B Algorithms and hyperparameters

Figure: Algorithm 2 Adaptive Soft Actor-Critic (AdaSAC). Modifications to SAC are marked in purple.

### B.1 Experimental setup

All environments are generated from the Gymansium library (Brockman et al., ). For the Atari experiments, we mimic the implementation choices made in Dopamine RL (Castro et al., ) since it is the only one to use the evaluation metric recommended by Machado et al. (). Namely, we use the game over signal to terminate an episode instead of the life signal. The input given to the neural network is a concatenation of $4$ frames in grayscale of dimension $84$ by $84$. To get a new frame, we sample $4$ frames from the Gym environment configured with no frame-skip, and we apply a max pooling operation on the $2$ last grayscale frames. We use sticky actions to make the environment stochastic (with $p=0.25$). The performance is the one obtained during training. We consider a different behavioral policy for the experiment with an infinite hyperparameter space. Instead of taking an $\epsilon$-greedy policy parameterized by $\epsilon_{b}$ as explained in Section 4.1, we sample each action from a probability distribution that is inversely proportional to the current loss value. That way, the problem of passive learning is further mitigated since the second or third most accurate networks will have a higher probability of being selected to sample actions. Importantly, when AdaDQN selects between different losses, the comparison is made between the values of the L2 losses even if the backpropagation is performed with another loss. The $3$ Atari games in the experiments of Section 5.3 are chosen from the $5$ games presented in Aitchison et al. (). The $6$ Atari games in the experiments of Section 5.4 are chosen from Franke et al. ().

Figure: Figure 9: AdaDQN and SEARL pipelines. AdaDQN and SEARL share the same exploitation and exploration strategies. They only differ in the way the $Q$-functions are trained and in the nature of the fitness value $f_{i}$. In both algorithms, the exploitation and exploration phase is performed every $M$ steps. It is referred to as the hyperparameter frequency (HP freq) in Figures 7 and 22.
Refer to caption: x9.png

AdaQN - SEARL comparison. The exploitation strategy of SEARL and AdaDQN is based on a tournament selection with elitism (Miller et al., ). The tournament comprises $K-1$ steps for a population of $K$ networks. At each step, $3$ networks are sampled uniformly without replacement, and the network with the best fitness value $f_{i}$ is qualified to populate the next generation. The last spot is reserved for the network with the best fitness value across the entire population. Then, every network present more than twice in the population undergoes a mutation operation. This is the exploration phase. One mutation operation out of the $5$ operations we consider (architecture change, activation function change, loss change, optimizer change, and learning rate change) is chosen with equal probability. When the architecture is changed, one of the following operations is selected $\{\pm 1$ CNN layer, $\pm 4$ CNN channels, $\pm 1$ kernel size, $\pm 1$ CNN stride, $\pm 1$ MLP layer, $\pm 16$ MLP neurons$\}$ with probability $[0.1,\frac{0.4}{3},\frac{0.4}{3},\frac{0.4}{3},0.1,0.4]$. Those probabilities are not tuned. They are set by hand so that the number of layers of the CNN and MLP do not change too often. If the hyperparameter goes outside the search space described in Table 4, the value is clipped to the boundary. In every case, the network weights are preserved as much as possible. This means that when the architecture changes, only the new weights that were not present before the mutation operation are generated. We choose to reset the optimizer each time a network undergoes a genetic modification. Instead of an elitism mechanism, we investigated, on Lunar Lander, a truncation mechanism (Jaderberg et al., ) where the $20\%$ most accurate networks are selected for the next generation. The performances are similar to the ones obtained with the elitism mechanism. We also tried to reset the weights after each generation. This did not lead to performance improvements. Therefore, we kept the proposed pipeline as close to SEARL as possible.

SEARL evaluates each network’s return empirically to estimate its performance. We set the minimum number of environment steps to evaluate each network to $M$ divided by the number of networks in the population so that at least $M$ new samples are collected in the replay buffer. This also synchronizes the frequency at which the hyperparameters are updated with AdaDQN. SEARL network weights are updated every $4$ training step, similarly to AdaDQN ($G=4$ in Table 4).

AdaQN - Meta-gradient RL comparison. The experiment comparing AdaDQN to Meta-gradient DQN is made with the default hyperparameters of DQN. Each network of AdaDQN is trained with a different discount factor, and the comparison between online networks is made by computing the approximation error parameterized by a cross-validated discount factor $\bar{\gamma}$. The optimizer used for the outer loss of Meta-gradient DQN is Adam with a learning rate of $0.001$ as in the original paper (Xu et al., ). Similarly to Xu et al. (), we set the size of the discount factor’s embedding space to $16$.

**Table 1: Summary of all fixed hyperparameters used for the Lunar Lander experiments (see Section 5.1).**
| Shared across all algorithms |  |
| --- | --- |
| Discount factor $\gamma$ | $0.99$ |
| Horizon | $1000$ |
| Initial replay buffer size | $1000$ |
| Replay buffer size | $10^{4}$ |
| Batch Size | $32$ |
| Target update frequency $T$ | $200$ |
| Training frequency $G$ | $1$ |
| Activation function | ReLU |
| Learning rate | $3\times 10^{-4}$ |
| Optimizer | Adam |
| Loss | L2 |
| Starting $\epsilon$ | $1$ |
| Ending $\epsilon$ | $0.01$ |
| Linear decay duration of $\epsilon$ | 1000 |
| AdaDQN |  |
| Starting $\epsilon_{b}$ | $1$ |
| Ending $\epsilon_{b}$ | $0.01$ |
| Linear decay duration of $\epsilon_{b}$ | Until end |

**Table 3: Summary of all fixed hyperparameters used for the Atari experiments with a finite hyperparameter space (see Section 5.3).**
| Shared across all algorithms |  |
| --- | --- |
| Discount factor $\gamma$ | $0.99$ |
| Horizon | $27000$ |
| Full action space | No |
| Reward clipping | clip$(-1,1)$ |
| Initial replay buffer size | $20000$ |
| Replay buffer size | $10^{6}$ |
| Batch Size | $32$ |
| Target update frequency $T$ | $8000$ |
| Training frequency $G$ | $4$ |
| Learning rate | $5\times 10^{-5}$ |
| Optimizer | Adam |
| Loss | L2 |
| Starting $\epsilon$ | $1$ |
| Ending $\epsilon$ | $0.01$ |
| Linear decay duration of $\epsilon$ | $250000$ |
| Vanilla DQN |  |
| Activation function | ReLU |
| $\epsilon_{\text{Adam}}$ | $1.5\times 10^{-4}$ |
| CNN n layers | $3$ |
| CNN channels | $[32,64,64]$ |
| CNN kernel size | $[8,4,3]$ |
| CNN stride | $[4,2,1]$ |
| MLP n layers | $1$ |
| MLP n neurons | $512$ |
| AdaDQN |  |
| Starting $\epsilon_{b}$ | $1$ |
| Ending $\epsilon_{b}$ | $0.01$ |
| Linear decay duration of $\epsilon_{b}$ | Until end |

## Appendix C Training time and memory requirements

We address time complexity by reporting the performances obtained in Section 5.2 and 5.4 with the clock time as the horizontal axis in Figure 10. On the left, AdaSAC still outperforms grid search but is slower than random search as AdaQN carries out several networks in parallel. We stress that this experiment is designed to demonstrate the capabilities of AdaSAC when a high number of hyperparameters is considered. When the number of network decreases to $5$, AdaQN becomes more competitive against random search as Figure 10 (right) testifies. In this case, AdaDQN outperforms all baselines in terms of training time and sample efficiency. We point out that Meta-gradient DQN, evaluated in Section 5.4, is $4\%$ slower to run than AdaDQN. Finally, we recall that, in this work, the main focus is on sample-efficiency, as it is the predominant bottleneck in practical scenarios.

Concerning the memory requirements, AdaSAC uses an additional $300$Mb from the vRAM for the MuJoCo experiments presented in Section 5.2 compared to SAC. AdaDQN uses $200$Mb more from the vRAM than random search for the Atari experiments presented in Section 5.4. We argue that those increases remain small compared to the benefit brought by AdaQN.

Figure: Figure 10: Performances observed in Section 5.2 (left) and Section 5.4 (right) according to the training time. When evaluated with a high number of hyperparameters to consider in parallel, AdaQN suffers from a slower training time but is still faster than grid search. When the number of hyperparameters is reasonable ($\approx 5$), AdaQN is competitive against all baselines.
Refer to caption: x10.png

## Appendix D Additional experiments

### D.1 On-the-fly hyperparameter selection on MuJoCo

Figure: Figure 11: Exhaustive legend of Figure 4 showing the ranking by AUC of the $16$ considered sets of hyperparameters. Interestingly, the ranking does not follow a predictable order, as there is no clear ordering between each hyperparameter. Importantly, the ranking changes between MuJoCo environments, justifying the usage of AdaSAC, which adapts the hyperparameter schedule for each environment and each seed.
Refer to caption: x11.png

### D.2 On-the-fly architecture selection on MuJoCo

Figure: Figure 12: On-the-fly architecture selection on MuJoCo. All architectures contain hidden layers with $256$ neurons. The architectures are indicated in the legend. Left: AdaSAC is more sample-efficient than grid search. Right: AdaSAC yields similar performances as the best-performing architectures. RandSAC and AdaSAC are on par.
Refer to caption: x12.png

Figure: Figure 13: Per environment return of AdaSAC and RandSAC when $4$ different architectures are given as input. Below each performance plot, a bar plot presents the distribution of the hyperparameters selected for computing the target across all seeds. AdaSAC effectively ignores the worst-performing architecture ($256$) while actively selecting the best-performing architecture. Interestingly, in HumanoidStandup, despite not performing well, AdaSAC selects the architecture $256,256$, which is the best-performing-architecture.
Refer to caption: x13.png

### D.3 On-the-fly learning rate selection on MuJoCo

Figure: Figure 14: On-the-fly learning rate selection on MuJoCo. The space of hyperparameters is composed of $3$ different learning rates. Left: AdaSAC is more sample-efficient than random search and grid search. Right: AdaSAC yields performances slightly above the best-performing learning rate. RandSAC and AdaSAC are on par.
Refer to caption: x14.png

Figure: Figure 15: Per environment return of AdaSAC and RandSAC when $3$ different learning rates are given as input. Below each performance plot, a bar plot presents the distribution of the hyperparameters selected for computing the target across all seeds. AdaSAC creates a custom learning rate schedule for each environment and each seed. Interestingly, this schedule seems to choose a low learning rate at the beginning of the training before increasing it later during the training.
Refer to caption: x15.png

### D.4 On-the-fly optimizer selection on MuJoCo

Figure: Figure 16: On-the-fly optimizer selection on MuJoCo. The space of hyperparameters is composed of $3$ different optimizers. Left: AdaSAC is more sample-efficient than random search and grid search. Right: AdaSAC yields similar performances as the best-performing optimizer (AdamW, Loshchilov & Hutter ()). RandSAC and AdaSAC are on par.
Refer to caption: x16.png

Figure: Figure 17: Per environment return of AdaSAC and RandSAC when $3$ different optimizers are given as input. Below each performance plot, a bar plot presents the distribution of the hyperparameters selected for computing the target across all seeds. AdaSAC effectively ignores RMSProp, which performs poorly in most environments.
Refer to caption: x17.png

### D.5 On-the-fly activation function selection on MuJoCo

Figure: Figure 18: On-the-fly activation function selection on MuJoCo. The space of hyperparameters is composed of $3$ different activation functions. Left: AdaSAC is more sample-efficient than random search and grid search. Right: AdaSAC performs slightly below the best-performing activation function. In this setting, RandSAC suffers from the fact that a network with Tanh activation functions is selected as the target network one-third of the time.
Refer to caption: x18.png

Figure: Figure 19: Per environment return of AdaSAC and RandSAC when $3$ different activation functions are given as input. Below each performance plot, a bar plot presents the distribution of the hyperparameters selected for computing the target across all seeds. Remarkably, AdaSAC selects the Sigmoid activation function in environments where this activation function seems beneficial (Ant, HalfCheetah, and Walker2d), while it does not select the Sigmoid activation function in environments where the individual run performs poorly (Humanoid and HumanoidStandup).
Refer to caption: x19.png

### D.6 On-the-fly hyperparameter selection on Atari

Figure: Figure 20: AdaDQN performs similarly to the best individual hyperparameters when selecting from $3$ activation functions (left), $3$ architectures (middle), or $3$ different values of Adam’s epsilon parameter (right). Below each performance plot, a bar plot presents the distribution of the hyperparameters selected for computing the target.
Refer to caption: x20.png

Figure: Figure 21: Per game performances of AdaDQN and the baselines when the space of hyperparameters is infinite. We believe that the wide range of returns is due to the fact that the hyperparameter space is large. Nevertheless, AdaDQN reaches higher IQM returns than SEARL on all games except on BattleZone, where only one seed performs well.
Refer to caption: x21.png

Figure: Figure 22: Per game performances of AdaDQN and SEARL at $40$M frames for different algorithm’s hyperparameter values. As each online network is evaluated in SEARL, we report the maximum and minimum observed return (SEARL - min and SEARL - max). Remarkably, AdaDQN’s performances are always above the minimum performances of SEARL. In $4$ out of the $6$ considered games, AdaDQN outperforms SEARL maximum return on all considered settings. High target freq corresponds to $T=8000$ and low target freq corresponds to $T=4000$. HP freq is noted $M$ in Figure 9, the high value is $M=80000$ and the low value is $M=40000$.
Refer to caption: x22.png

Figure: Figure 23: Evolution of AdaDQN’s hyperparameters along the training for one seed on Pong. While the online networks explore a vast portion of the hyperparameter space, the target network focuses on a few values.
Refer to caption: x23.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Can AdaQN effectively be applied on continuous action spaces (actor-critic frameworks) especially in complex high-dim environments? Which Q should be use to derive a policy network (maybe similar as we choose the one for target calculation)?
soundness: 3
strengths: 1. Simple but novel and effective design of training multiple Q functions with different hyper-parameters settings and target selection mechanism which means no additional samples are required for hyper-parameter tuning. 
2. AdaQN is shown to performance better in Mujoco and Atari benchmarks than DQN and other autoRL baselines
3. Theoretical support is also presented to support the motivation of the selection mechanism in AdaQN.
4. AdaQN can be applied to other critic-based algorithms
summary: The authors have developed a new AutoRL approach called Adaptive Q-Network (AdaQN), which aims to address the non-stationary optimization procedure in RL without requiring additional samples. The algorithm learns multiple Q-functions with different hyper-parameters while using the smallest approximation error as a shared target. So there's no need for additional samples. The paper also provides theoretical motivation that supports the algorithm and through experiments on Mujoco and Atari, AdaQN is shown to have better performance and sample efficiency.
weaknesses: From my understanding, AdaQN's performance is still dependent on the initial range of hyper-parameters considered in the multiple Q setup. If this range is not representative of the optimal settings, AdaQN may not achieve the best possible performance. Also, it seems error or bias in the calculation of approximation error will affect the selection of the shared target.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Can you give more explanation on the evaluation protocol?

What is the performance when running the optimal hyperparameters from the start?

Can you clarify the parts that I listed in the weaknesses, including the discussion on the discount factor?
soundness: 3
strengths: - The main idea is clear, theoretically justified and seems like a good idea.
- The performance improves significantly.
- The experiments are thorough, and look at several metrics, include ablations, etc.
- The exposition is good.
summary: This work proposes a new AutoRL algorithms that constructs an ensemble of Q-networks, each learned with different hyperaparameters (e.g., different learning rates). The main novelty is that during learning, there is an interaction between the different Q-networks; namely, the best performing Q-network is used for the target computation for all networks. This means that the individual performances of the networks will be different, potentially better than the performance obtained by just learning by scratch with a chosen hyperparameter. Moreover, the best performing hyperparameters may also change as learning progresses, highlighting an attractive property of the proposal. Action selection is also performed by picking the best performing Q-network from the previous transition, together with an $\epsilon$ random selection of a Q-network from the ensemble. Experiments are performed on DQN lunar lander as well as SAC on MuJoCo tasks. They compare against other AutoRL algorithms like SEARL and DEHB, and obtain superior performance. The work also includes ablation studies on the components of the algorithm, e.g., the epsilon parameter, and the Q-network selection mechanism. Moreover, they provide other evidence, such as directly plotting which networks were selected. The final performances on the tasks seem reasonable, and the learning speed including hyperparameter selection is good compared to the baselines.
weaknesses: - The evaluation protocol was only briefly discussed, and was not completely clear to me. Particularly, how the baselines results are plotted could have been explained more prominently. I believe these plots include the hyperparameter selection cost on x-axis, but this could be clearer. Moreover, for reference, it would be good to know what the performance would be when running from scratch with the best performing hyperparameters.

In particular, what does this mean:
> Additionally, we focus on sample efficiency; thus, for reporting the grid search results, we multiply the number of interactions of the best achieved performance by the number of individual trials, as advocated by Franke et al. (2021). We also report the average performance over the individual runs as the performance that a random search would obtain in expectation.

This bit seems very important for interpreting the results, but it's only briefly mentioned. I would suggest explaining this with equation notation to make it precise. Moreover, perhaps it would be useful to explain the hyperparameter tuning task setting, what the objective and metrics are in the background. Moreover, the axis labels such as "Env steps" can be confusing, because the same notation is used in standard RL where one just runs the best performing hyperparameters without considering cost of searching the hyperparameters. These points would be good to clarify.

- Around the end of the paper, there was a part about also selecting the $\gamma$ discount parameter, but this seemed unsound to me. The discount parameter affects the Q-function, so different Q-functions would necessarily require different targets, and I don’t think they could be shared. For example setting $\gamma=0$ would probably give good small prediction errors, but would not actually perform well.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Given the motivation to "speed up hyperparameters selection," how can the selected hyperparameters by AdaQN be used effectively? Can the selected hyperparameters by AdaQN be used to train a better DQN? It seems the improvements by AdaQN are due to the use of different Q-functions, similar to ensemble methods, with the difference being the use of various hyperparameters instead of just different initializations. Could the paper be motivated by performance improvement using different hyperparameters rather than speeding up hyperparameter selection?
soundness: 3
strengths: This paper addresses the crucial issue of hyperparameter selection in deep RL and provides an effective solution. The proposed method is well written and organized. Comprehensive experiments validate the effectiveness of the method.
summary: This paper addresses the sensitivity of hyperparameters in deep Reinforcement Learning (RL) and proposes a method to speed up hyperparameter selection compared to previous AutoRL approaches. The authors introduce the Adaptive Q-Network (AdaQN), which learns several Q-functions, each trained with different hyperparameters. These Q-functions are updated online using the one with the smallest approximation error as a shared target. Comprehensive experiments on MuJoCo and Atari environments demonstrate the effectiveness of AdaQN.
weaknesses: Theorem 4.1 explains why the empirical Bellman operator in Eq (3) derived from experience replay is unbiased to the true Bellman operator in Eq (2). However, it lacks a theoretical analysis on why the selection scheme from Eq (2) leads to convergence rather than local minima. A formal theoretical analysis would strengthen the paper by explaining the convergence properties of the proposed method.

Meta-review: The paper presents an autoRL technique to dynamically and automatically select the best hyperparameters in temporal different Q-network updates.  The strength of this work is an effective technique that clearly improves the stability and sample efficiency of learning.  The main weakness of this work is the computational overhead of updating an ensemble of Q-networks.  However, this computation can be easily parallelized when there is sufficient computational resources.  The approach is well motivated.  It is simple and yet effective.  It clearly advances the state of the art.

Decision: Accept (Poster)