Dynamic Learning Rate for Deep Reinforcement Learning: A Bandit Approach

## Abstract

Abstract In Deep Reinforcement Learning models trained using gradient-based techniques, the choice of optimizer and its learning rate are crucial to achieving good performance: higher learning rates can prevent the model from learning effectively, while lower ones might slow convergence. Additionally, due to the non-stationarity of the objective function, the best-performing learning rate can change over the training steps. To adapt the learning rate, a standard technique consists of using decay schedulers. However, these schedulers assume that the model is progressively approaching convergence, which may not always be true, leading to delayed or premature adjustments. In this work, we propose dynamic Learning Rate for deep Reinforcement Learning (LRRL), a meta-learning approach that selects the learning rate based on the agent’s performance during training. LRRL is based on a multi-armed bandit algorithm, where each arm represents a different learning rate, and the bandit feedback is provided by the cumulative returns of the RL policy to update the arms’ probability distribution. Our empirical results demonstrate that LRRL can substantially improve the performance of deep RL algorithms.

## 1 Introduction

Reinforcement Learning (RL), when combined with function approximators such as Artificial Neural Networks (ANNs), has shown success in learning policies that outperform humans in complex games by leveraging extensive datasets (see, *e.g.*, , , , ). While ANNs were previously used as value function approximators [], the introduction of Deep Q-Networks (DQN) by [, ] marked a significant breakthrough by improving learning stability through two mechanisms: the target network and experience replay.

The experience replay (see ) stores the agent’s interactions within the environment, allowing sampling of past interactions in a random way that disrupts their correlation. The target network further stabilizes the learning process by periodically copying the parameters of the learning network. This strategy is crucial because the Bellman update —using estimations to update other estimations— would otherwise occur using the same network, potentially causing divergence. By leveraging the target network, gradient steps are directed towards a periodically fixed target, ensuring more stability in the learning process. Additionally, the learning rate hyperparameter controls the magnitude of these gradient steps in optimizers such as the stochastic gradient descent algorithm, affecting the training convergence.

The learning rate is one of the most important hyperparameters, with previous work demonstrating that decreasing its value during policy finetuning can enhance performance by up to 25% in vanilla DQN []. Determining the appropriate learning rate(^1^11The terms “learning rate” and “step-size” are often used interchangeably in the literature and they technically refer to the same concept.) is essential for achieving good model performance: higher values can prevent the agent from learning, while lower values can lead to slow convergence (see , , ). However, finding a learning rate value that improves the model performance requires extensive and computationally expensive testing. In order to adapt its initial choice during training, optimizers such as Adam [] and RMSProp [] employ an internal scheme that dynamically adjusts the learning rate, considering, for instance, past gradient information. Nevertheless, various learning rate scheduling strategies can be combined with the optimizer to decrease the learning rate and improve the convergence over the training steps.

Standard learning rate schedulers typically decrease the learning rate based on training progress using, *e.g.*, linear or exponential decay strategies [, ]. In the context of RL, this approach can lead to premature or delayed learning rate adjustments, which may hinder the agent’s ability to learn. Unlike supervised learning, RL usually involves generating data by trading off between exploration (discovery of new states) and exploitation (refining of the agent’s knowledge). As the policy improves, the data distribution encountered by the agent becomes more concentrated, but this evolution occurs at a different pace than the overall training progress. For instance, some environments require extensive exploration due to the sparseness of rewards, while others need more exploitation to refine the policy to the complexity of the task. Consequently, a more sophisticated decaying learning rate strategy that accounts for policy performance rather than training steps can significantly enhance learning in deep RL.

In this work, we propose dynamic Learning Rate for deep Reinforcement Learning (LRRL), a method to select the learning rate *on the fly* for deep RL. Our approach acknowledges that *different learning phases require different learning rates*, and as such, instead of scheduling the learning rate decay using some blanket approach, we dynamically choose the learning rate using a Multi-Armed Bandit (MAB) algorithm, which accounts for the current policy’s performance. Our method has the advantage of being algorithm-agnostic and applicable to any optimizer, although the results show that it works best when coupled with Adam. We conduct experiments on our approach using baselines provided in the Dopamine framework (see ). Our results focus on exploiting different settings for LRRL to illustrate its robustness under many possible configurations. Our main contributions are the following:

- •
We introduce LRRL, the first approach, to our knowledge, that leverages a multi-armed bandit algorithm to select the learning rate dynamically in deep RL. Our results demonstrate that LRRL achieves competitive performance with or superior to standard deep RL algorithms using fixed baselines or traditional learning rate schedulers.
- •
Our results show that LRRL significantly reduces the need for hyperparameter optimization by dynamically selecting from a set of possible learning rates using a multi-armed bandit approach. This method mitigates the need for exhaustive techniques like grid search, as it efficiently adapts the learning rate during training in a single run, rather than requiring multiple runs to test each learning rate individually.
- •
We assess the robustness of our method by employing the Adam and RMSProp optimizers with different sets of arms. We also compare the results using stochastic and adversarial multi-armed bandit algorithms in Appendix A.2.

## 2 Related Work

#### Multi-armed bandit for (hyper)parameter selection.

Deep RL is known to be overly optimistic in the face of uncertainty [], and many works have addressed this issue by proposing conservative policy updates [, , ]. However, when the agent is able to interact with the environment, this optimism can encourage exploration, potentially leading to the discovery of higher returns. Building on this idea, [] use an adversarial MAB algorithm to trade-off between pessimistic and optimistic policy updates based on the agent’s performance over the learning process. In order to select the learning rate for stochastic gradient MCMC, [] employ an algorithm based on Successive Halving [, ], a MAB strategy that promotes promising arms and prunes suboptimal ones over time. In the context of hyperparameter optimization, Successive Halving has also been used in combination with infinite-arm bandits to select hyperparameters for supervised learning [, ]. A key difference between these approaches to hyperparameter optimization for supervised learning and our work is that we focus on selecting the best learning rate from a predefined set, rather than performing an extensive and computationally expensive search over the hyperparameter space. Close to our work, [] propose Adam with Bandit Sampling (ADAMBS), which employs the Exponential-weight algorithm for Exploration and Exploitation [] to enhance sample efficiency by incorporating importance sampling within the Adam optimizer. While ADAMBS prioritizes informative samples, our method focuses on RL tasks by dynamically adjusting the learning rate, accelerating learning when the policy is far from optimal, and slowing it down as it converges.

#### Learning rate adapters/schedulers.

Optimizers such as RMSProp []
have an adaptive mechanism to update a set of parameters $\theta$ by normalizing past gradients, while Adam [] also incorporates momentum to smooth gradient steps. However, despite their widespread adoption, these algorithms have inherent limitations in non-stationary environments since they do not adapt to changes in the objective function over time (see  ).
Increment-Delta-Bar-Delta (IDBD), introduced by  [], has an adaptive mechanism based on the loss to adjust the learning rate $\eta_{i}$ for each sample $x_{i}$ for linear regression and has been extended to settings including RL [].
Learning rate schedulers with time decay [, ] are coupled with optimizers, assuming gradual convergence to a good solution, but often require task-specific manual tuning. A meta-gradient reinforcement learning is proposed in [], composed of a two-level optimization process: one that uses the agent’s objective and the other to learn meta-parameters of the objective function. Our work differs from these methods by employing a multi-armed bandit approach to dynamically select the learning rate over the training process, specifically targeting RL settings.

## 3 Preliminaries

This section introduces the Reinforcement Learning and Multi-Arm Bandits frameworks, defining supporting notation.

### 3.1 Deep Reinforcement Learning

An RL task is defined by a Markov Decision Process (MDP), that is by a tuple $\mathcal{(S,A},P,R,\gamma,T)$, where $\mathcal{S}$ denotes the state space, $\mathcal{A}$ the set of possible actions, $P:\mathcal{S\times A\times S\rightarrow}[0,1]$ the transition probability, $R:\mathcal{S\times A}\rightarrow\mathbb{R}$ the reward function, $\gamma\in[0,1]$ the discount factor, and $T$ the horizon length in episodic settings (see, *e.g.*,  for details). In RL, starting from an initial state $s_{0}$, a learner called *agent* interacts with the environment by picking, at time $t$, an action $a_{t}$ depending on the current state $s_{t}$. In return, it receives a reward $r_{t}=R(s_{t},a_{t})$, reaching a new state $s_{t+1}$ according to the transition probability $P(s_{t},a_{t},\,\cdot\,)$. The agent’s objective is to learn a policy $\pi:\mathcal{S\times A}\rightarrow[0,1]$ which maps a distribution of actions given the current state, aiming to maximize expected returns $Q^{\pi}(s,a)=\mathbb{E}_{\pi}\bigl{[}\sum_{t=0}^{T}\gamma^{t}r_{t}\mid s_{0}=s
,a_{0}=a\bigr{]}$.

To learn how to perform a task, value function-based algorithms coupled with ANNs [, ] approximate the *quality* of a given state-action pair $Q(s,a)$ using parameters $\theta$ to derive a policy $\pi_{\theta}(s)=\arg\max_{a\in\mathcal{A}}Q_{\theta}(s,a)$. By storing transitions $(s,a,r,s^{\prime})=(s_{t},a_{t},r_{t},s_{t+1})$ into the replay memory $\mathcal{D}$, the objective is to minimize the loss function $\mathcal{J}(\theta)$ defined by:

$$ $\mathcal{J}(\theta)=\mathbb{E}_{(s,a,r,s^{\prime})\sim\mathcal{D}}\Bigl{[}r+ \gamma\,\max_{a^{\prime}\in\mathcal{A}}Q_{{\theta}^{-}}(s^{\prime},a^{\prime}) -Q_{\theta}(s,a)\Bigr{]}^{2}\,,$ (1) $$

where $\theta^{-}$ are the target network parameters used to compute the target of the learning network $y=r+\gamma\,\max_{a^{\prime}\in\mathcal{A}}Q_{{\theta}^{-}}(s^{\prime},a^{
\prime})$. The parameters $\theta^{-}$ are periodically updated by copying the parameters $\theta$, leveraging stability during the learning process by fixing the target $y$. The minimization, and hence, the update of the parameters $\theta$, is done according to the optimizer’s routine. A simple possibility is to use stochastic gradient descent using mini-batch approximations of the loss gradient:

$$ $\theta_{n+1}\leftarrow\theta_{n}-\eta\nabla_{\theta}\mathcal{J}(\theta_{n})\,, \quad\text{where}\quad\nabla_{\theta}\mathcal{J}(\theta_{n})\approx\frac{1}{| \mathcal{B}|}\sum_{(s,a,r,s^{\prime})\in\mathcal{B}}2\bigl{(}Q_{\theta}(s,a)-y \bigr{)}\nabla_{\theta}Q_{\theta}(s,a)$ (2) $$

with $\mathcal{B}$ being a mini-batch of transitions sampled from $\mathcal{D}$ and $\eta$ is a single scalar value called the *learning rate*. Unlike in supervised learning, where the loss function $\mathcal{J}(\theta)$ is typically stationary, RL presents a fundamentally different challenge: the policy is continuously evolving, leading to shifting distributions of states, actions, and rewards over time. This continuous evolution introduces instability during the learning process, which deep RL mitigates by employing a large replay memory and calculating the target using a frozen network with parameters $\theta^{-}$. However, stability also depends on how the parameters $\theta$ change during each update. This work aims to control these changes by dynamically selecting the learning rate $\eta$ over the training steps.

### 3.2 Multi-Armed Bandit

Multi-Armed Bandits (MAB) provide an elegant framework for making sequential decisions under uncertainty (see for instance ). MAB can be viewed as a special case of RL with a single state, where at each round $n$, the agent selects an arm $k_{n}\in\{1,\ldots,K\}$ from a set of $K$ arms and receives a feedback (reward) $f_{n}(k_{n})\in\mathbb{R}$. Like RL, MAB algorithms must balance the trade-off between exploring arms that have been tried less frequently and exploiting arms that have yielded higher rewards up to time $n$.

To account for the non-stationarity of the RL rewards, we will consider in this work the MAB setting of adversarial bandits []. In this setting, at each round $n$, the agent selects an arm $k_{n}$ according to some distribution $p_{n}$ while the environment (the *adversary*) arbitrarily (*e.g.*, without stationary constraints) determines the rewards $f_{n}(k)$ for all arms $k\in\mathcal{K}$. MAB algorithms are designed to minimize the *pseudo-regret* $G_{N}$ after $N$ rounds defined by:

$$ $G_{N}=\max_{k\in\{1,\dots,K\}}\,\mathbb{E}\Biggl{[}\sum_{n=1}^{N}f_{n}(k)-\sum _{n=1}^{N}f_{n}(k_{n})\Biggr{]}\,,$ $$

where the randomness of the expectation depends on the MAB algorithm and on the adversarial environment, $\smash{\sum_{n=1}^{N}f_{n}(k)}$ represents the accumulated reward of the single best arm in hindsight, and $\smash{\sum_{n=1}^{N}f_{n}(k_{n})}$ is the accumulated reward obtained by the algorithm. A significant component in the adversarial setting is to ensure that each arm $k$ has a non-zero probability $p_{n}(k)>0$ of being selected at each round $n$: this guarantees exploration, which is essential for the algorithm’s robustness to environment changes.

## 4 Dynamic Learning Rate for Deep RL

In this section, we tackle the challenge of selecting the learning rate over the training steps by introducing a dynamic Learning Rate for deep Reinforcement Learning (LRRL). LRRL is a meta-learning approach designed to dynamically select the learning rate in response to the agent’s performance. LRRL couples with stochastic gradient descent optimizers and adapts the learning rate based on the reward achieved by the policy $\pi_{\theta}$ using an adversarial MAB algorithm. As the agent interacts with the environment, the average of observed rewards is used as bandit feedback to guide the selection of the most appropriate learning rate throughout the training process.

### 4.1 Selecting the Learning Rate Dynamically

Our problem can be framed as selecting a learning rate $\eta$ for policy updates —specifically, when updating the parameters $\theta$ after $\lambda$ interactions with the environment.

Before training, a set $\mathcal{K}=\{\eta_{1},\dots,\eta_{K}\}$ of $K$ learning rates are defined by the user.
Then, during training, a MAB algorithm selects, at every round $n$ —that is, at every $\kappa$ interactions with the environment— an arm $k_{n}\in\{1,\dots,K\}$ according to a probability distribution $p_{n}$ defined based on previous rewards, as explained in the next section. The parameters $\theta$ are then updated using the sampled learning rate $\eta_{k_{n}}$. The steps involved in this meta-learning approach are summarized in Algorithm 1.

Figure: Algorithm 1 dynamic Learning Rate for deep Reinforcement Learning (LRRL)

Note that the same algorithm might be used with learning rates schedulers, that is with $\mathcal{K}=\{\eta_{1},\dots,\eta_{K}\}$ where $\eta_{k}:\mathbb{N}\to\mathbb{R}_{+}$ is a predefined function, usually converging towards 0 0 at infinity. If so, the learning rate used at round $n$ of the optimization is $\eta_{k_{n}}(n)$.

### 4.2 Updating the Probability Distribution

As we expect that the agent’s performance —and hence, the cumulative rewards— will improve over time, the MAB algorithm should receive non-stationary feedback. To take this non-stationary nature of the learning into account, we employ the Exponential-weight algorithm for Exploration and Exploitation (Exp3, see  for an introduction). At round $n$, Exp3 chooses the next arm (and its associated learning rate) according to the arm probability distribution $p_{n}$ which is based on weights $(w_{n}(k))_{1\leq k\leq K}$ updated recursively. Those weights incorporate a time-decay factor $\delta\in(0,1]$ that increases the importance of recent feedback, allowing the algorithm to respond more quickly to improvements in policy performance.

Specifically, after picking arm $k_{n}$ at round $n$, the RL agent interacts $C$ times with the environment and the MAB algorithm receives a feedback $f_{n}$ corresponding to the average reward of those $C$ interactions. Based on [], this feedback is then used to compute the *improvement in performance*, denoted by $f^{\prime}_{n}$, obtained by subtracting the average of the past $j$ bandit feedbacks from the most recent one $f_{n}$:

$$ $f_{n}^{\prime}=f_{n}-\frac{1}{j}\sum_{i=0}^{j-1}f_{n-i}\,.$ $$

The improvement in performance allows computation of the next weights $w_{n+1}$ as follows, where initially $w_{1}=(0,\ldots,0)$:

$$ $\forall k\in\{1,\ldots,K\},\quad w_{n+1}(k)=\begin{cases}\delta\,w_{n}(k)+ \alpha\,\frac{f^{\prime}_{n}}{e^{w_{n}(k)}}&\text{if }k=k_{n}\\ \delta\,w_{n}(k)&\text{otherwise}\,,\end{cases}$ (3) $$

where $\alpha>0$ is a step-size parameter. The distribution $p_{n+1}$, used to draw the next arm $k_{n+1}$, is

$$ $\forall k\in\{1,\ldots,K\},\quad p_{n+1}(k)=\frac{e^{w_{n+1}(k)}}{\sum_{k^{ \prime}=1}^{K}e^{w_{n+1}(k^{\prime})}}\,.$ (4) $$

This update rule ensures that as the policy $\pi_{\theta}$ improves, the MAB algorithm continues to favor learning rates that are most beneficial under the current policy performance, thereby effectively handling the non-stationarity inherent in the learning process.

## 5 Experiments

In the following sections, we investigate whether combining LRRL with Adam or RMSProp —two widely used optimizers— can improve cumulative returns in deep RL algorithms. To assess this, we compare LRRL against learning methods with and without schedulers using the baseline implementation of DQN provided in Dopamine []. We test LRRL under different configurations and Atari games, reporting the average and standard deviation of returns over 5 runs. Details on the evaluation metrics and hyperparameters used in these experiments are summarized in Appendix B.

### 5.1 Comparing LRRL with Standard Learning

In our first experiment, we consider a set of $5$ learning rates, and compare the performance of 5 configurations of LRRL, each of them using a subset of those learning rates, against the DQN algorithm reaching best performance, in terms of maximum average return, among the $5$ possible learning rates choices (see Figure 5 in Appendix A.1). More precisely, the set of learning rates is

$$ $\mathcal{K}(5)=\bigl{\{}1.5625\times 10^{-5},3.125\times 10^{-5},6.25\times 10 ^{-5},1.25\times 10^{-4},2.5\times 10^{-4}\bigr{\}}\,.$ $$

The whole set $\mathcal{K}(5)$ is used by one LRRL version, while others are based on the $3$ lowest ($\mathcal{K}_{\text{lowest}}(3))$, $3$ middle ($\mathcal{K}_{\text{middle}}(3))$, $3$ highest ($\mathcal{K}_{\text{highest}}(3)$) and 3 taking the lowest/middle/highest ($\mathcal{K}_{\text{sparse}}(3)$) learning rate values. All experiments use the Adam optimizer, and we report the return based on the same number of environment iterations.

Results are gathered in Table 1 and illustrated in Figure 1 along with four Atari games. They show that LRRL outperforms standard learning in two out of four tasks while remaining competitive in the others. Notably, LRRL not only matches or exceeds performance but also reduces the need for extensive parameter tuning by incorporating multiple learning rates in a single run. However, as the results indicate, the $3$-arm bandit variants exhibit different behaviors during the learning process, suggesting that the choice of the number of arms and their values still requires task-specific tuning to achieve good performance.

Figure: Figure 1: A comparison between 5 configurations of LRRL with various learning rate subsets and the DQN algorithm reaching the best performance among possible learning rates.
Refer to caption: extracted/5928492/results_arms.png

**Table 1: Max average return (best in bold if significantly better than others) and its standard deviation for 4 Atari games.**
| Game | DQN | LRRL $\mathcal{K}_{\text{lowest}}(3)$ | LRRL $\mathcal{K}_{\text{middle}}(3)$ | LRRL $\mathcal{K}_{\text{highest}}(3)$ | LRRL $\mathcal{K}_{\text{sparse}}(3)$ | LRRL $\mathcal{K}(5)$ |
| --- | --- | --- | --- | --- | --- | --- |
| Asteroids | 1 085 $\pm$ 63 | 1 065 $\pm$ 70 | 1 079 $\pm$ 48 | 1 061 $\pm$ 59 | 1007 $\pm$ 127 | 1 085 $\pm$ 88 |
| Breakout | 217 $\pm$ 14 | 220 $\pm$ 24 | 201 $\pm$ 6 | 177 $\pm$ 8 | 270 $\pm$ 13 | 253 $\pm$ 20 |
| Pong | 19 $\pm$ 0.4 | 18 $\pm$ 0.3 | 19 $\pm$ 0.9 | 19 $\pm$ 0.5 | 19 $\pm$ 0.5 | 19 $\pm$ 0.8 |
| Seaquest | 5 881 $\pm$ 1 533 | 5 905 $\pm$ 2 557 | 6 135 $\pm$ 2 229 | 6 231 $\pm$ 1 802 | 8 920 $\pm$ 2 759 | 6 799 $\pm$ 2 060 |

To illustrate how LRRL adapts during training in response to non-stationary bandit feedback as policy performance improves, Figure 2 shows the systematic sampling of pulled arms (learning rates) and corresponding returns over training steps from a single run of LRRL $\mathcal{K}(5)$. In most of the tested environments, LRRL behaves similarly to time-decay schedulers by selecting higher learning rates during the early stages of training, gradually shifting toward arms with lower rates as training progresses. The exception is in the Pong environment, where the model converges after only a few iterations, resulting in a more uniform probability distribution across the set of arms.

Figure: Figure 2: Systematic sampling of normalized learning rates and returns over the training steps using LRRL $\mathcal{K}(5)$ with Adam optimizer, through a single run. For each episode, we show the selected learning rate using different colors.
Refer to caption: extracted/5928492/results_convergence.png

### 5.2 Combining and Comparing Schedulers with LRRL

Next, we consider using LRRL combined with learning rate schedulers. Specifically, we employ schedulers with exponential decay rate of the form $\eta(n)=\eta_{0}\times e^{-d\,n}$, where $\eta_{0}$ is a fixed initial value (common to each scheduler and equal to $6.25\times 10^{-5}$ in our experiment), $d$ is the exponential decay rate and $n$ is the number of policy updates (*i.e.*, of MAB rounds). We define a set of $3$ schedulers $\mathcal{K}_{s}$, where each arm represents a scheduler using a different decay rate $d=\{1,2,3\}\times 10^{-7}$, and compare the results of LRRL with each scheduler individually, using the Adam optimizer.

Figure 3 and Table 2 show that LRRL combined with schedulers can substantially increase final performance compared to using exponential decay schedulers for some environments while remaining competitive for others. The dashed black line represents the max average return achieved by Adam without learning rate decay, resulting in slightly worse performance compared to using schedulers, aligning with findings in previous work by [], who linearly decay the learning rate to 0 0.

Figure: Figure 3: A comparison between LRRL with arms as schedulers and schedulers individually.
Refer to caption: extracted/5928492/results_schedulers.png

**Table 2: Max average return (best in bold) and its standard deviation for 4 Atari games.**
| Game | DQN | $d=1\times 10^{-7}$ | $d=2\times 10^{-7}$ | $d=3\times 10^{-7}$ | LRRL ($\mathcal{K}_{s}$) |
| --- | --- | --- | --- | --- | --- |
| Asteroids | 1 028 $\pm$ 53 | 1 025 $\pm$ 52 | 1 013 $\pm$ 56 | 1 063 $\pm$ 82 | 1 015 $\pm$ 33 |
| Breakout | 144 $\pm$ 12 | 149 $\pm$ 11 | 151 $\pm$ 7 | 144 $\pm$ 12 | 233 $\pm$ 19 |
| Seaquest | 5 881 $\pm$ 1 533 | 5 284 $\pm$ 1 134 | 5 793$\pm$ 1 225 | 6 612 $\pm$ 835 | 13 864 $\pm$ 3 581 |
| Video Pinball | 410 186 $\pm$ 193 328 | 388 768 $\pm$ 195 150 | 450 015 $\pm$ 178 876 | 362 645 $\pm$ 172 169 | 388 308 $\pm$ 103 862 |

### 5.3 RMSProp Optimizer and more Environments

Another widely used optimizer for training deep RL models is RMSProp, which, like Adam, features an adaptive learning rate mechanism. Adam builds upon RMSProp by retaining exponential moving averages to give more weight to recent gradients while incorporating momentum. Although the standard RMSProp does not feature momentum, we found that adding momentum to RMSProp can increase both the performance of DQN and LRRL, aligning with findings in the literature [, ].

In the following experiment, we compare the performance of RMSProp with Nesterov’s momentum (RMSProp-M), and Adam when coupled with either LRRL or the best-performing single learning rate when using DQN. As shown in Figure 4 and Table 3, LRRL coupled with Adam consistently outperforms our configuration using RMSProp-M and the baseline using standard DQN. Moreover, LRRL (RMSProp-M) underperforms compared to DQN without LRRL in two out of three tasks due to its slow convergence despite better jumpstart performance. Future work should investigate whether this slow convergence is linked to factors such as the environment’s stochasticity or the optimizer’s features such as the absence of bias correction in the first and second moment estimates.

Figure: Figure 4: A comparison between Adam and RMSProp with momentum, using either DQN or LRRL.
Refer to caption: extracted/5928492/results_rms.png

**Table 3: Max average return (best in bold) and its standard deviation for 3 Atari games.**
| Game | DQN (RMSProp-M) | LRRL (RMSProp-M) | DQN (Adam) | LRRL (Adam) |
| --- | --- | --- | --- | --- |
| Asterix | 11 464 $\pm$ 2 848 | 6 499 $\pm$ 927 | 12 561 $\pm$ 1 245 | 15 017 $\pm$ 3 892 |
| Ms. Pacman | 3 301 $\pm$ 310 | 2 696 $\pm$ 248 | 3 232 $\pm$ 114 | 3 310 $\pm$ 231 |
| Space Invaders | 1 490 $\pm$ 132 | 2 712 $\pm$ 73 | 2 874 $\pm$ 319 | 3 641 $\pm$ 1 030 |

## 6 Conclusion

In this work, we introduced dynamic Learning Rate for Deep Reinforcement Learning (LRRL), a meta-learning approach for selecting the optimizer’s learning rate on the fly. We demonstrated empirically that combining LRRL with the Adam optimizer could significantly enhance the performance of the value-based algorithm DQN, outperforming baselines and learning rate schedulers in some tasks while remaining competitive in others. Furthermore, by employing a multi-armed bandit algorithm, LRRL reduces the need for extensive hyperparameter tuning, as it explores a set of learning rates in a single run with minimal extra computational overhead.

While this work focused on dynamically selecting the best-performing learning rate, future investigations could extend LRRL ideas to other critical hyperparameters, such as mini-batch size, which also plays a key role in the model’s convergence. Moreover, although LRRL selects learning rates based on policy performance, alternative feedback mechanisms could be explored, such as using gradient information to select block-wise (e.g., per-layer) learning rates, extending these ideas to supervised learning and applying them to other non-stationary objective functions, including those encountered in Continual Learning [, , ].

## Acknowledgment

This work was granted access to the HPC resources of IDRIS under the allocation 2024-A0171015651 made by GENCI. HD and FF acknowledge funding from Inria Project WOMBAT. LFS is supported by a Discovery Early Career Researcher Award from the Australian Research Council (DE240101190)

## Appendix A Supplementary Experiments

### A.1 Baseline Evaluation with Varying Learning Rates

To establish a baseline to compare learning without our approach LRRL, we run individual arm values as baseline learning rate using the Adam optimizer. The results presented in Figure 5 align with common expectations by showing that higher learning rates fail to learn for most environments while lower ones can lead to the worst jumpstart performance and slow convergence.

Figure: Figure 5: DQN performance using Adam optimizer with varying learning rates across 4 Atari games.
Refer to caption: extracted/5928492/results_baselines_adam.png

### A.2 A Comparison between Multi-Armed Bandits Algorithms

Adversarial MAB algorithms are designed for environments where the reward distribution changes over time. In contrast, stochastic MAB algorithms assume that rewards are drawn from fixed but unknown probability distributions. To validate our choice and address our method’s robustness, we compare Exp3 with the stochastic MAB algorithm MOSS (Minimax Optimal Strategy in the Stochastic case) []. MOSS trade-off exploration-exploitation by pulling the arm $k$ with highest upper confidence bound given by:

$$ $B_{k}(n)=\hat{\mu}_{k}(n)+\rho\,\sqrt{\frac{\max\Bigl{(}\log{\frac{n}{Kn_{k}(n )}},0\Bigr{)}}{n_{k}(n)}}$ $$

where $\hat{\mu}_{k}(n)$ is the empirical average reward for arm $k$ and $n_{k}(n)$ is the number of times it has been pulled up to timestep $t$.

In Figure 6, we use different bandit step-sizes $\alpha$ for Exp3 and parameter $\rho$, which balance exploration and exploitation in MOSS. Additionally, the bandit feedback used in MOSS is the average cumulative reward $\frac{R}{C}$. The results indicate that while Exp3 performs better overall, MOSS can still achieve competitive results depending on the amount of exploration, demonstrating the robustness of our approach regarding the MAB algorithm employed.

Figure: Figure 6: A comparison between adversarial and stochastic MAB algorithms.
Refer to caption: extracted/5928492/results_mab.png

## Appendix B Experiments Details

### B.1 Evaluation Terminology

In this section, we describe the evaluation metrics that can be used to evaluate agent’s performance as it interacts with an environment. Based on Dopamine, we use the evaluation step-size “iterations”, which is defined as a predetermined number of episodes. Figure 7 illustrates the evaluation metrics used in this work, as defined in []:

- •
Max average return: The highest average return obtained by an algorithm throughout the learning process. It is calculated by averaging the outcomes across multiple individual runs.
- •
Final performance: The performance of an algorithm after a predefined number of interactions. While two algorithms may reach the same final performance, they might require different amounts of data to do so. This metric captures the efficiency of an algorithm in reaching a certain level of performance within a limited number of interactions. In Figure 7, the final performance overlaps with the max average return, represented by the black dashed line.
- •
Jumpstart performance: The performance at the initial stages of training, starting from a policy with randomized parameters $\theta$. In Figure 7, Algorithm B exhibits better jumpstart performance but ultimately achieves lower final performance than Algorithm A. A lower jumpstart performance can result from factors such as a lower learning rate, although this work demonstrates that this does not necessarily lead to worse final performance.

Figure: Figure 7: Performance curves of two RL algorithms (adapted from taylor2009transfer []).
Refer to caption: extracted/5928492/evaluation.png

### B.2 Further Experimental Details

In the following, we list the set of arms, optimizer and the bandit step-size used in each experiment.

#### Section 5.1 – Comparing LRRL with Standard Learning.

- •
Optimizer: Adam
- •
Bandit step-size: $\alpha=0.2$
- •
Considered sets of learning rates:
$\displaystyle\mathcal{K}(5)$
$\displaystyle=\bigl{\{}1.5625\times 10^{-5},3.125\times 10^{-5},6.25\times 10^
{-5},1.25\times 10^{-4},2.5\times 10^{-4}\bigr{\}}$
$\displaystyle\mathcal{K}_{\text{lowest}}(3)$
$\displaystyle=\bigl{\{}1.5625\times 10^{-5},3.125\times 10^{-5},6.25\times 10^
{-5}\bigr{\}}$
$\displaystyle\hskip 25.03856pt\mathcal{K}_{\text{middle}}(3)=\bigl{\{}3.125
\times 10^{-5},6.25\times 10^{-5},1.25\times 10^{-4}\bigr{\}}$
$\displaystyle\hskip 84.50471pt\mathcal{K}_{\text{highest}}(3)=\bigl{\{}6.25
\times 10^{-5},1.25\times 10^{-4},2.5\times 10^{-4}\bigr{\}}$
$\displaystyle\mathcal{K}_{\text{sparse}}(3)$
$\displaystyle=\bigl{\{}1.5625\times 10^{-5},\phantom{3.125\times 10^{-5},}\,6.
25\times 10^{-5},\phantom{1.25\times 10^{-4},}\,2.5\times 10^{-4}\bigr{\}}$

#### Section 5.2 – Combining and Comparing Schedulers with LRRL.

- •
Optimizer: Adam
- •
Bandit step-size: $\alpha=0.2$
- •
Initial learning rate of schedulers: $\eta_{0}=6.25\times 10^{-5}$

#### Section 5.3 – RMSProp Optimizer and More Environments.

- •
Optimizer: RMSProp for baselines
- •
Bandit step-size: $\alpha=0.2$
- •
LRRL set of learning rates: $\mathcal{K}=\bigl{\{}1.5625\times 10^{-5},3.125\times 10^{-5},6.25\times 10^{-
5}\bigr{\}}$

#### Section A.2 – A Comparison between Multi-Armed Bandit Algorithms.

- •
Optimizer: Adam
- •
MOSS/Exp3 set of learning rates: $\mathcal{K}=\bigl{\{}3.125\times 10^{-5},6.25\times 10^{-5},1.25\times 10^{-4}
\bigr{\}}$

### B.3 Hyperparameters

In this section, we outline the hyperparameters used in the experiments. The optimizers from the Optax library [] are employed alongside the JAX [] implementation of the DQN algorithm [, ], as provided by the Dopamine framework [].

**Table 4: Hyperparameters**
| Hyperparameter | Setting |
| --- | --- |
| Sticky actions | True |
| Sticky actions probability | 0.25 |
| Discount factor ($\gamma$) | 0.99 |
| Frames stacked | 4 |
| Mini-batch size ($\mathcal{B}$) | 32 |
| Replay memory start size | 20000 |
| Learning network update rate ($\lambda$) | 4 steps |
| Minimum environment steps ($\kappa$) | 1 episode |
| Target network update rate ($\tau$) | 8000 steps |
| Initial exploration ($\epsilon$) | 1 |
| Exploration decay rate | 0.01 |
| Exploration decay period | 250000 steps |
| Environment steps per iteration | 250000 steps |
| Reward clipping | [-1, 1] |
| Network neurons per layer | 32, 64, 64 |
| Hardware | V100 GPU |
| Adam hyperparameters |  |
| $\beta_{1}$ decay | 0.9 |
| $\beta_{2}$ decay | 0.999 |
| Eps | 1.5e-4 |
| RMSProp hyperparameters |  |
| Decay | 0.9 |
| Momentum (if *True*) | 0.999 |
| Centered | False |
| Eps | 1.5e-4 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Tables 1 and 2 which report max average returns seem unecessary given that the entire learning curves are also presented.
Also, reporting the maximum return is generally not a good practice due to the additional noise in the estimate. See [1] for a more in-depth discussion of this. 

- In section 5.1, which optimizer is used with DQN? Is it RMSProp, Adam or a different choice?

- In fig.1, what are "# iterations" on the x-axis? How many training steps are done in total?

- When formulating the meta-learning problem as a bandit problem, the formulation does not make use of the fact that different learning rates are related. In this paper, the learning rates are treated as separate discrete actions while they are in fact continuous quantities. Similar learning rates should have similar optimization properties. 
Perhaps using a formulation that allows some generalization between the bandit actions (learning rates) could be useful here. i.e. linear bandits or more general contextual bandits. 


- Because of the way the reward is structured, it seems like sparse rewards would be a problem for these methods. 
If a feedback window happens to be in the middle of the episode where no rewards are given, then it would make it seem like that meta-action was not useful.
That could explain why in Pong, which has a relatively sparse reward, the agent does not manage to adapt the learning rate much and ends up mostly picking learning rates relatively uniformly.

- As a comment: In the background section (lines 177 onward), it is mentioned that the adversarial bandit framework deals with nonsationarity, but this is not quite accurate. For that, you would need to consider sequences of changing best actions, not just one action in hindsight. For a more detailed discussion, see Ch 31.1 from [2]. I would suggest rewording this section a little.
This does not negatively impact the algorithm in the end since the actual bandit algorithm used does have a decay parameter which does try to account for the nonstationarity by emphasizing recent experiences. 


[1] "Deep Reinforcement Learning that Matters" Henderson et al.

[2] "Bandit Algorithms" Lattimore and Szepesvari (Chapter 31.1)  https://tor-lattimore.com/downloads/book/book.pdf
soundness: 3
strengths: The paper presents the proposed algorithm quickly and the motivation is clear. 
I find that the formulation of the meta-learning problem as a bandit problem is interesting and it is nice to see this direction explored more in this paper.

There are some interesting insights into which learning rates are favourable for different environments by looking at the ones chosen by the proposed algorithm (in Fig.2).
summary: This paper proposes a meta-learning algorithm, LRRL, which selects the learning rate throughout the training process using a bandit algorithm. This algorithm is tested in with DQN on deep RL benchmark environments and in conjunction with differnet optimizers such as RMSProp and Adam. Some analysis is done to assess the learning rate choices through learning.
weaknesses: Generally, my main concern are the purported benefits of the algorithm.
The original motivation is that learning rates can be important to tune in general and also throughout training. 
From the results (e.g. Fig.1), it looks like the choice of the set of learning rates for LRRL is important. 
Many of the settings do not lead to any benefit over the baseline agent. Moreover, simply choosing the largest set of learning rates, $\mathcal{K}(5)$, does not necessarily lead to any improvements.
This runs counter to the original motivation of reducing hyperparameter sensitivity to the learning rate.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: * How would the authors address the concern of having to tune the set of learning rates for Exp3?

* Would Exp3 be able to encode the intuition I described above in some way? How would the authors reconcile this potential drawback?
soundness: 3
strengths: * The authors show that LRRL achieves competitive performance compared to standard deep RL algorithms that use fixed learning rates or traditional learning rate schedulers. They test the algorithm using the DQN algorithm with the Adam optimizer on a variety of Atari games.
* LRRL reduces the need for hyperparameter optimization in principle, but see below for my concern about this.
summary: The paper proposes to use a non-stationary version of Exp3 (a multi-armed bandit algorithm) to dynamically adjust the learning rate in deep reinforcement learning models during training. The algorithm begins by initializing a set of learning rates. During training, the Exp3 algorithm selects a learning rate from the set based on the agent’s recent performance. The agent then interacts with the environment for a fixed number of steps using the selected learning rate, before the process repeats.
weaknesses: * The choice of the number of learning rate arms and their values still requires task-specific tuning. This is evidenced in Table 1, where LRRL seems to have significantly different performance depending on what choice of learning rates are used. I would have expected the bandit algorithm to be relatively robust to its set of arms, but this is not the case. 
    * K_sparse(3) performs far better than the other choices. In my view, this approach seems to just shift hyperparameter tuning to a different parameter.

* One intuition for decay of learning rates is that the optimizer takes large steps toward the optimal policy first (perhaps incurring less reward), but later on "hones in" on the best policy using a smaller learning rate (incurring high reward). The bandit approach wouldn't be able to encode this kind of intuition, where you sacrifice some reward initially so that higher rewards can be attained later on. I think this is a main drawback of this approach.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 4
questions: Why was Pong swapped out for Video PinBall in section 5.2? How did the paper choose which Atari games to use for each section? Did the paper look at all Atari environments but reported those with a significant result in the main body? If so, this should be made clear in the main body.
soundness: 2
strengths: * Clear Writing, easy to follow and understand
* Good empirical description. Almost all the necessary details about how the experiment was run were included in the main body or Appendix B
summary: The paper presents a bandit algorithm for selecting the learning rate of DQN in a set of Atari games. The paper shows how the bandit-selected learning rates over one fixed set of learning rates can, in some cases, yield better performance than the default DQN learning rate.
weaknesses: * The biggest weakness is the range of learning rates tested. The paper uses a set of 5 learning rates, all within one order of magnitude, to test LRRL. Most deep RL algorithms are highly sensitive to learning rates. While these are known to be a good range for DQN, as it has been around for over a decade, it is difficult to say whether LRRL equipped with this range will work well with other new algorithms we want to deploy this one. Additionally, would the Adam vs RMSProp results still hold if different ranges of learning rates were given? Two suggestions could address this:
1. Different sets of learning rates for LRRLs.
     - A wider range; Like something that goes from 10^-1 to 10^-6
     - Randomly generated ranges. Maybe have uniform or log uniform sampling over some range 10^-1 to 10^-6, and see how LRRL works with 5 LRs in this range
2. Try with other algorithms (see next bullet point)

* The other big concern with this paper is that it makes claims for Deep Reinforcement Learning Algorithms but only provides evidence on one (DQN). Two fixes could be either:
     - Try LRRL with learning rates on other algorithms: PPO, SAC, A3C, TD3, DDQN etc.
     - Re-adjust the claims in the paper to make it clear that you are studying LRRL+DQN. This would include changes in wording throughout and changing the title to Dynamic Learning Rate for DQN.

* Since the experiments section uses 5 independent runs to measure the average return, we would recommend a different uncertainty measure for the reported results:
     - A bootstrapped confidence interval or a tolerance interval that could be representative of the variability in performance across different seeds and the same learning rate. This would be useful in figure 1.
    - 95% confidence intervals. Although this comes with an assumption that the returns for your given agent+learning rate is normally distributed.
    - The meaning shaded region was not explicitly mentioned any where. I assume it is std dev?

* It would have been useful to see how a bandit algorithm over this set K(5) would compare to other hyperparameter approaches such as bayesian optimization packages like Optuna, which is  commonly used for dynamically tuning learning rates and other deep reinforcement learning hyperparameters.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: See above.
soundness: 2
strengths: Overall, adjusting the learning rate during training is an important problem. The proposed approach is conceptually straightforward, using multi-armed bandits, which is also relatively easy to implement.
summary: This work presents Dynamic Learning Rate for Reinforcement Learning (LRRL), a meta-learning approach that uses a multi-armed bandit algorithm to dynamically adjust the learning rate of an RL agent during training. Each arm of the bandit represents a different learning rate, and the bandit's feedback mechanism uses the cumulative returns of the RL policy to update the probability distribution over the arms. The authors argue that this approach addresses the non-stationarity of the objective function in RL, where the optimal learning rate can change throughout training, and claim that it reduces the need for extensive hyperparameter tuning compared to traditional learning rate decay schedules. Empirical results on several Atari games using the DQN algorithm are presented, comparing LRRL against fixed baselines, traditional schedulers, and alternative optimizers.
weaknesses: The main idea in the paper for using bandits for hyperparameter selection in RL is not new. Multiple prior works have explored similar ideas, using bandits to adapt various aspects of learning, including exploration strategies and policy updates. The authors' attempt to position LRRL as a distinct approach within the context of meta-learning is also not entirely convincing, as meta-gradient methods already address the adaptation of learning parameters (see e.g., Flennerhag, Sebastian, et al., among many others). 

Re experiments: The scope of the experiments in the paper is limited. The authors focus on a single RL algorithm (DQN) and a restricted set of environments. I propose the authors run their approach using sota RL algorithms, as well as continuous control tasks, which would help understand the broader applicability of LRRL. Finally, the authors should include a comparison of their approach to sota meta-gradient method.

The paper could also benefit from a more detailed discussion of the theoretical properties of Exp3 in the context of learning rate adaptation and a more comprehensive exploration of alternative bandit algorithms (e.g., linear contextual bandits).

Meta-review: This work proposes a meta learning approach to learning rates selection for reinforcement learning algorithms based on the Multi Armed Bandits model. Although it was agreed that this approach holds promise for the problem of hyperparameter selection, the reviewers also agreed that the experimental evaluation of this work is insufficient to show substantiated promise of the proposed approach. We encourage the authors to expand this evaluation to larger and more convincing scenarios. Additionally, this work would benefit from drawing connections with other bandit literature works such as online model selection techniques that can also be used to solve the problem tackled by the authors.

Decision: Reject