Adaptive $Q$-Network: On-the-fly Target Selection for Deep Reinforcement Learning

Authors: Théo Vincent, Fabian Wahren, Jan Peters, Boris Belousov, Carlo D'Eramo
Primary area: reinforcement learning
Keywords: automated reinforcement learning, deep reinforcement learning, hyperparameter selection

Abstract:
Deep Reinforcement Learning (RL) is well known for being highly sensitive to hyperparameters, requiring practitioners substantial efforts to optimize them for the problem at hand. This also limits the applicability of RL in real-world scenarios. In recent years, the field of automated Reinforcement Learning (AutoRL) has grown in popularity by trying to address this issue. However, these approaches typically hinge on additional samples to select well-performing hyperparameters, hindering sample-efficiency and practicality. Furthermore, most AutoRL methods are heavily based on already existing AutoML methods, which were originally developed neglecting the additional challenges inherent to RL due to its non-stationarities. In this work, we propose a new approach for AutoRL, called _Adaptive $Q$-Network_ (AdaQN), that is tailored to RL to take into account the non-stationarity of the optimization procedure without requiring additional samples. AdaQN learns several $Q$-functions, each one trained with different hyperparameters, which are updated online using the $Q$-function with the smallest approximation error as a shared target. Our selection scheme simultaneously handles different hyperparameters while coping with the non-stationarity induced by the RL optimization procedure and being orthogonal to any critic-based RL algorithm. We demonstrate that AdaQN is theoretically sound and empirically validate it in MuJoCo control problems and Atari $2600$ games, showing benefits in sample-efficiency, overall performance, robustness to stochasticity and training stability. Our code is available at *https://github.com/theovincent/AdaDQN*.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730516347805

thread_type:
official_review

<thread_object>
creator: reviewer_pnat

<content>
summary:
This paper addresses the sensitivity of hyperparameters in deep Reinforcement Learning (RL) and proposes a method to speed up hyperparameter selection compared to previous AutoRL approaches. The authors introduce the Adaptive Q-Network (AdaQN), which learns several Q-functions, each trained with different hyperparameters. These Q-functions are updated online using the one with the smallest approximation error as a shared target. Comprehensive experiments on MuJoCo and Atari environments demonstrate the effectiveness of AdaQN.

strengths:
This paper addresses the crucial issue of hyperparameter selection in deep RL and provides an effective solution. The proposed method is well written and organized. Comprehensive experiments validate the effectiveness of the method.

weaknesses:
Theorem 4.1 explains why the empirical Bellman operator in Eq (3) derived from experience replay is unbiased to the true Bellman operator in Eq (2). However, it lacks a theoretical analysis on why the selection scheme from Eq (2) leads to convergence rather than local minima. A formal theoretical analysis would strengthen the paper by explaining the convergence properties of the proposed method.

questions:
Given the motivation to "speed up hyperparameters selection," how can the selected hyperparameters by AdaQN be used effectively? Can the selected hyperparameters by AdaQN be used to train a better DQN? It seems the improvements by AdaQN are due to the use of different Q-functions, similar to ensemble methods, with the difference being the use of various hyperparameters instead of just different initializations. Could the paper be motivated by performance improvement using different hyperparameters rather than speeding up hyperparameter selection?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730806960848

thread_type:
official_review

<thread_object>
creator: reviewer_kj2w

<content>
summary:
This work proposes a new AutoRL algorithms that constructs an ensemble of Q-networks, each learned with different hyperaparameters (e.g., different learning rates). The main novelty is that during learning, there is an interaction between the different Q-networks; namely, the best performing Q-network is used for the target computation for all networks. This means that the individual performances of the networks will be different, potentially better than the performance obtained by just learning by scratch with a chosen hyperparameter. Moreover, the best performing hyperparameters may also change as learning progresses, highlighting an attractive property of the proposal. Action selection is also performed by picking the best performing Q-network from the previous transition, together with an $\epsilon$ random selection of a Q-network from the ensemble. Experiments are performed on DQN lunar lander as well as SAC on MuJoCo tasks. They compare against other AutoRL algorithms like SEARL and DEHB, and obtain superior performance. The work also includes ablation studies on the components of the algorithm, e.g., the epsilon parameter, and the Q-network selection mechanism. Moreover, they provide other evidence, such as directly plotting which networks were selected. The final performances on the tasks seem reasonable, and the learning speed including hyperparameter selection is good compared to the baselines.

strengths:
- The main idea is clear, theoretically justified and seems like a good idea.
- The performance improves significantly.
- The experiments are thorough, and look at several metrics, include ablations, etc.
- The exposition is good.

weaknesses:
- The evaluation protocol was only briefly discussed, and was not completely clear to me. Particularly, how the baselines results are plotted could have been explained more prominently. I believe these plots include the hyperparameter selection cost on x-axis, but this could be clearer. Moreover, for reference, it would be good to know what the performance would be when running from scratch with the best performing hyperparameters.

In particular, what does this mean:
> Additionally, we focus on sample efficiency; thus, for reporting the grid search results, we multiply the number of interactions of the best achieved performance by the number of individual trials, as advocated by Franke et al. (2021). We also report the average performance over the individual runs as the performance that a random search would obtain in expectation.

This bit seems very important for interpreting the results, but it's only briefly mentioned. I would suggest explaining this with equation notation to make it precise. Moreover, perhaps it would be useful to explain the hyperparameter tuning task setting, what the objective and metrics are in the background. Moreover, the axis labels such as "Env steps" can be confusing, because the same notation is used in standard RL where one just runs the best performing hyperparameters without considering cost of searching the hyperparameters. These points would be good to clarify.

- Around the end of the paper, there was a part about also selecting the $\gamma$ discount parameter, but this seemed unsound to me. The discount parameter affects the Q-function, so different Q-functions would necessarily require different targets, and I don’t think they could be shared. For example setting $\gamma=0$ would probably give good small prediction errors, but would not actually perform well.

questions:
Can you give more explanation on the evaluation protocol?

What is the performance when running the optimal hyperparameters from the start?

Can you clarify the parts that I listed in the weaknesses, including the discussion on the discount factor?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730870399162

thread_type:
official_review

<thread_object>
creator: reviewer_fxwz

<content>
summary:
The authors have developed a new AutoRL approach called Adaptive Q-Network (AdaQN), which aims to address the non-stationary optimization procedure in RL without requiring additional samples. The algorithm learns multiple Q-functions with different hyper-parameters while using the smallest approximation error as a shared target. So there's no need for additional samples. The paper also provides theoretical motivation that supports the algorithm and through experiments on Mujoco and Atari, AdaQN is shown to have better performance and sample efficiency.

strengths:
1. Simple but novel and effective design of training multiple Q functions with different hyper-parameters settings and target selection mechanism which means no additional samples are required for hyper-parameter tuning. 
2. AdaQN is shown to performance better in Mujoco and Atari benchmarks than DQN and other autoRL baselines
3. Theoretical support is also presented to support the motivation of the selection mechanism in AdaQN.
4. AdaQN can be applied to other critic-based algorithms

weaknesses:
From my understanding, AdaQN's performance is still dependent on the initial range of hyper-parameters considered in the multiple Q setup. If this range is not representative of the optimal settings, AdaQN may not achieve the best possible performance. Also, it seems error or bias in the calculation of approximation error will affect the selection of the shared target.

questions:
Can AdaQN effectively be applied on continuous action spaces (actor-critic frameworks) especially in complex high-dim environments? Which Q should be use to derive a policy network (maybe similar as we choose the one for target calculation)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732100862697

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Additions to the revised submission

comment:
We thank the Reviewers for their valuable feedback which helped us to improve the paper. We have submitted a revision with two additions highlighted in orange:
1. *[requested by Reviewer PnAT]* We prove the convergence of AdaQN to the optimal action-value function in the tabular setting in Appendix A.1.

2. *[requested by Reviewer Kj2W]* We explain the evaluation protocol in detail in Appendix B.2. For each method, we describe the steps leading to the reported performances in Section $5$.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents an autoRL technique to dynamically and automatically select the best hyperparameters in temporal different Q-network updates.  The strength of this work is an effective technique that clearly improves the stability and sample efficiency of learning.  The main weakness of this work is the computational overhead of updating an ensemble of Q-networks.  However, this computation can be easily parallelized when there is sufficient computational resources.  The approach is well motivated.  It is simple and yet effective.  It clearly advances the state of the art.