# Multi-parameter Control for the (1+(L,L))-GA on OneMax via Deep Reinforcement Learning

Tai Nguyen, Phong Le, Carola Doerr, Nguyen Dang

► To cite this version:

Tai Nguyen, Phong Le, Carola Doerr, Nguyen Dang. Multi-parameter Control for the (1+(L,L))-GA on OneMax via Deep Reinforcement Learning. Foundations of Genetic Algorithms XVIII (FOGA), ACM/SIGEVO, 2025, Leiden, Netherlands. 10.1145/3729878.3746703 . hal-05171396

HAL Id: hal-05171396

https://hal.sorbonne-universite.fr/hal-05171396v1

Submitted on 19 Jul 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Multi-parameter Control for the $(1 + (\lambda, \lambda))$-GA on OneMax
via Deep Reinforcement Learning

Tai Nguyen

University of St Andrews
St Andrews, United Kingdom
Sorbonne Université, CNRS, LIP6
Paris, France

Phong Le

University of St Andrews
St Andrews, United Kingdom

Carola Doerr

Sorbonne Université, CNRS, LIP6
Paris, France

Nguyen Dang

University of St Andrews
St Andrews, United Kingdom

## Abstract

It is well known that evolutionary algorithms can benefit from dynamic choices of the key parameters that control their behavior, to adjust their search strategy to the different stages of the optimization process. A prominent example where dynamic parameter choices have shown a provable super-constant speed-up is the $(1 + (\lambda, \lambda))$ Genetic Algorithm optimizing the OneMax function. While optimal parameter control policies result in linear expected running times, this is not possible with static parameter choices. This result has spurred a lot of interest in parameter control policies. However, many works, in particular theoretical running time analyses, focus on controlling one single parameter. Deriving policies for controlling multiple parameters remains very challenging. In this work, we reconsider the problem of the $(1 + (\lambda, \lambda))$ Genetic Algorithm optimizing OneMax. We decouple its four main parameters and investigate how well state-of-the-art deep reinforcement learning techniques can approximate good control policies. We show that although making deep reinforcement learning learn effectively is a challenging task, once it works, it is very powerful and is able to find policies that outperform all previously known control policies on the same benchmark. Based on the results found through reinforcement learning, we derive a simple control policy that consistently outperforms the default theory-recommended setting by 27% and the irace-tuned policy, the strongest existing control policy on this benchmark, by 13%, for all tested problem sizes up to 40,000.

## CCS Concepts

* Computing methodologies → Randomized search.

## 1 Introduction

Parameter control, studying how to best adjust the parameters of an optimization algorithm during its execution, is a topic that has gained significant interest in the evolutionary computation community in the last decades, both from an empirical angle [2, 25, 32] and from a running time analysis perspective [18]. Much of this interest was spurred by results showing that algorithms using a dynamic setup can significantly outperform their static counterparts. However, identifying optimal or reasonably good control policies remains a very challenging endeavor.

For the theory community, an important example that spurred much of the recent interest in parameter control is the observation that the $(1 + (\lambda, \lambda))$ Genetic Algorithm (GA)<fnref n="1" /> equipped with a one-fifth-success rule has an expected linear running time on ONEMAX; a performance that is impossible to obtain with static parameter settings [17]. Although such self-adjusting setting was proven beneficial, it remains limited to controlling a single parameter of the algorithm, the offspring population size $\lambda$. A decoupling of the key parameters of the $(1+(\lambda,\lambda))$-GA has been considered in [14] and, in a different way, in [4]. In the approach of Dang and Doerr [14], the policy still controls one parameter, the offspring population size $\lambda_m$ used in the mutation phase, whereas the offspring size of the crossover phase $\lambda_c = \gamma\lambda_m$, the mutation rate $p = \alpha\lambda/n$, and the crossover bias $c = \beta/\lambda_c$ are parameterized in static dependence of this controlled parameter. The work Antipov et al. [4] uses a different approach and samples, in each iteration, the values of the three parameters $\lambda_m, p$, and $c$ from heavy-tailed distributions.

In this work, we take an alternative approach to discover multi-parameter control policies for the $(1+(\lambda,\lambda))$-GA on ONEMAX. Instead of regarding the traditional parameter control setting, where parameter values have to be chosen “on the fly” (or “online”, as one would refer to in a Machine Learning context), i.e., while optimizing the problem at the same time, we move our focus to the *Dynamic Algorithm Configuration* setting [1, 10], where state-dependent parameter settings are learned through a dedicated training phase. In this context, we investigate the potential of leveraging (deep) reinforcement learning (RL) to support finding effective multi-parameter control policies for the $(1+(\lambda,\lambda))$-GA on ONEMAX. In fact, RL fits very well with parameter control contexts where the aim is to learn a policy that maps from the current state of the algorithm (during the optimization process) to the best parameter values for that state. Deep RL has become increasingly popular in algorithmic communities for learning parameter control policies in evolutionary algorithms (e.g., [10, 40, 48, 52]), for adaptively selecting components and adjusting them in hyper-heuristics (e.g., [57, 58]), and for dynamically selecting heuristics in AI planning solvers (e.g., [49]).

Despite their potentials, adopting deep-RL approaches in DAC context is non-trivial, as illustrated in recent studies on deep-RL learning stability using theory-derived DAC benchmarks [10, 42].

<footnote n="1">¹We are aware that the reference to *genetic algorithm* in the name may be ambiguous, but since the algorithm is well established, we prefer to refer to it by its original name.</footnote>
Previous theory-derived deep-RL studies were limited to single-parameter control settings, while in this work, we address a multi-parameter control problem using the (1+(λ,λ))-GA on ONEMAX. Referring the interested reader to [12] for a detailed analysis of the problem landscape, we point out that even for this comparatively simple algorithm optimizing the artificial ONEMAX problem, the structure of the parameter control policy problem is non-trivial and imposes challenges when learning in the corresponding DAC landscape. We emphasize that our core interest is in the control problem, not in the optimization of the ONEMAX problem. Put differently, we use this benchmark because it is well explored and offers a well-motivated baseline against which we can compare.

Our main contributions can be summarized as follows:

(1) We leverage Double Deep Q-Network (DDQN) [41], a commonly used deep RL approach in dynamic algorithm configuration contexts [42, 48, 49], to control four key parameters of the (1+(λ,λ))-GA. As most, but not all [5, 6, 11, 26] previous works on the (1+(λ,λ))-GA, we consider the well-explored setting of minimizing the expected running time of the (1+(λ,λ))-GA on ONEMAX. We demonstrate the challenges of applying DDQN in this context and highlight three key designs to enable effective learning, including: (i) a well-chosen representation of the action space; (ii) reward shifting; and (iii) an increased discount factor. With all these components in place, our DDQN-based approach is able to learn policies that outperform all previously known control policies.

(2) Based on the results suggested by DDQN, we derive a fairly simple and interpretable yet extremely effective parameter control policy for (1+(λ,λ))-GA on ONEMAX. Our derived policy significantly outperforms both theoretical and empirical work on parameter control. The effectiveness and the simplicity of our derived policy opens up the potential for new theoretical findings on this benchmark.

With these contributions, we expect to address researchers from at least two different domains. Theory-interested researchers may find our derived control policies and the ablation studies valuable. Empirically-oriented researchers with interest in RL may be interested in the challenges of adopting deep RL in algorithmic contexts and the insights provided by our findings on the impact of those key designs in DDQN's learning ability.

**Reproducibility:** Our code and data are available at [43].

# 2 Background

## 2.1 The ONEMAX Problem

We consider the problem of minimizing the average running time of the (1+(λ,λ))-GA on ONEMAX, a setting that has received considerable attention in the running time analysis community over the last years [3, 4, 8, 17, 19].

Below, we refer to ONEMAX as the original ONEMAX function ONEMAX: {0, 1}ⁿ → ℝ, x ↦ ∑ⁿᵢ=₁ xᵢ, but, as well known, the performance of the (1+(λ,λ))-GA is strictly identical on all ONEMAX-like functions ONEMAXₓ: {0, 1}ⁿ → ℝ, x ↦ |{1 ≤ i ≤ n|xᵢ = zᵢ}, z ∈ {0, 1}, since they are isomorphic to the original ONEMAX = ONEMAX(1,...,1) and the (1+(λ,λ))-GA is an unbiased

algorithm in the sense of Lehre and Witt proposed in [36]. All results presented below therefore naturally apply to all generalized ONEMAXₓ functions, sometimes referred to as *Mastermind* with two colors [21].

## 2.2 The (1+(λ,λ))-GA

Algorithm 1 presents a generalized version of the (1+(λ,λ))-GA, originally proposed in [19], with *four parameters* to be controlled, namely the *population sizes* $\lambda_m \in \mathbb{N}$ (for the mutation phase) and $\lambda_c \in \mathbb{N}$ (for the crossover phase), the *mutation rate coefficient* $\alpha \in \mathbb{R}_{>0}$, and the *crossover bias coefficient* $\beta \in \mathbb{R}_{>0}$. We briefly explain the different phases and parameters in the following paragraphs.

Starting from an initial solution, the (1+(λ,λ))-GA runs iteratively until an optimal solution is found.<fnref n="2" /> In each iteration, the algorithm has two main phases: the mutation phase and the crossover phase.

**Mutation phase.** In the mutation phase, $\lambda_m$ offspring are generated by applying standard bit mutation to the current-best solution $x$. More precisely, a mutation strength $\ell$ is sampled from the binomial distribution $Bin_{>0}(n, p)$ that is conditioned on returning a value $\ell > 0$ through rejection sampling; i.e., $\ell$ is sampled from $Bin(n, p)$ until a positive value is returned. Following the recommendations made in [17, 19], the *mutation rate* $p$ is parametrized as $p = \alpha\lambda_m/n$. Each of the $\lambda_m$ offspring $x^{(i)}$ is then obtained by applying the *flip$_\ell$* operator to $x$, which flips exactly $\ell$ bits in $x$, chosen uniformly at random.

**Crossover phase.** The best among the $\lambda_m$ solutions obtained in the mutation phase, referred to as $x'$, is selected for the crossover phase; in case of ties a random one among the best offspring is chosen. It is worth noting that $x'$ may (and frequently will be) worse than the current incumbent solution $x$. The goal of the crossover phase is to *repair* it, see [19] for a detailed intuition behind the (1+(λ,λ))-GA. In a nutshell, the crossover acts as a *genetic repair mechanism* that mitigates the potential loss of beneficial genes from the parent due to the *aggressive* mutation rate while preserving them in the offspring. To this end, $\lambda_c$ offspring are generated by combining $x'$ with its parent $x$ through the biased crossover operator $cross_c$, which creates a new search point by choosing, independently for each position $1 \le i \le n$, the entry of the second input with probability $c$ and choosing the entry from the first argument otherwise. We refer to $c$ as the *crossover bias*, and we parametrize $c = \beta/\lambda_c$, again following the suggestions made in [17, 19]. As an implementation detail, also in line with experiments made in previous work [3, Section 4.1], we only evaluate solutions $y^{(i)}$ that are different from both, $x$ and $x'$. This implies that the number of evaluations in this crossover phase can be smaller than $\lambda_c$. Finally, we note that the use of the crossover as repair mechanism is specific to this algorithm; crossover can have very different roles in evolutionary algorithms.

**Selection.** The best of all offspring, $y$ (selected again uniformly at random among all best options in case of multiple best offspring), replaces the parent $x$ if it is at least as good as it.

<footnote n="2">²In practice, one would of course use or add a second stopping criterion; but, as common in running time analysis, we only consider time to optimum here in this work.</footnote>
**Algorithm 1:** The $(1 + (\lambda, \lambda))$-GA with four parameters to be controlled, including the population sizes $\lambda_m$ (for the mutation phase) and $\lambda_c$ (for the crossover phase), the mutation rate coefficient $\alpha$, and the crossover bias coefficient $\beta$.

**Input:** Problem size $n$, fitness function $f(\cdot)$

1. $x \leftarrow$ a sample from $\{0, 1\}^n$ chosen uniformly at random;

2. **while** $x$ is not the optimal solution **do**

3. Choose $\lambda_m, \lambda_c, \alpha$ and $\beta$ based on $f(x)$

4. **Mutation phase:**

5. $p = \alpha\lambda_m/n$; ▶ mutation rate

6. Sample $\ell$ from $\text{Bin}_{>0}(n, p)$;

7. **for** $i = 1, \dots, \lambda_m$ **do** $x^{(i)} \leftarrow \text{flip}_\ell(x)$; Evaluate $f(x^{(i)})$;

8. Choose $x' \in \{x^{(1)}, \dots, x^{(\lambda_m)}\}$ with
$f(x') = \max\{f(x^{(1)}), \dots, f(x^{(\lambda_m)})\}$ u.a.r.;

9. **Crossover phase:**

10. $c = \beta/\lambda_c$; ▶ crossover bias

11. $\mathcal{Y} \leftarrow \emptyset$;

12. **for** $i = 1, \dots, \lambda_c$ **do**

13. $y^{(i)} \leftarrow \text{cross}_c(x, x')$;

14. **if** $y^{(i)} \notin \{x, x'\}$ **then** Evaluate $f(y^{(i)})$; Add $y^{(i)}$ to $\mathcal{Y}$;

15. **Selection and update step:**

16. Choose $y \in \{x'\} \cup \mathcal{Y}$ with
$f(y) = \max\{f(x'), \max_{y' \in \mathcal{Y}} f(y')\}$ u.a.r.;

17. **if** $f(y) \ge f(x)$ **then** $x \leftarrow y$;

The self-adjusting $(1+(\lambda,\lambda))$-GA differs from Algorithm 1 in that we update the offspring population size $\lambda_m$ at the end of each iteration. If a better solution is found, the population size is increased to $A\lambda_m$, for some static update strength $A > 1$. The population size is updated to $\beta\lambda_m$ otherwise, with $0 < \beta < 1$ being the second update strengths. To obtain a one-fifth-success rule, one sets $A = F^{(1/4)}$ and $b = 1/F$ for some fixed factor $F > 1$. With this setting, $\lambda_m$ remains constant if on average one out of five iterations is successful in finding a strictly better solution. This is the version investigated in [16, 19], with a fixed setting of $\alpha = \beta = \gamma = 1$ in the notation of Algorithm 2.

In an attempt to automatize the identification of good parameter control policies, Chen et al. [12] considered Algorithm 1 with $\lambda_c = \lambda_m$ and $\alpha = \beta = 1$ to investigate how well the automated algorithm configuration tool irace [39] would approximate a good policy for controlling $\lambda_m$. While a naïve application of irace was unable to find good policies, an iterative “binning-and-cascading” approach was able to learn control policies that are significantly better than all previously known policies. However, the iterative configuration approach is computationally expensive, leading Nguyen et al. [42] to investigate the effectiveness of employing deep reinforcement learning (RL), particularly DDQN [53], to dynamically control $\lambda_m$, resulting not only in highly effective control policies but also a significant improvement in the sampling efficiency of the learning step compared to [12].

## 2.4 Multi-parameter Control Policies

As already pointed out, Algorithm 2 has more parameters than the algorithm originally proposed in [17, 19]. This generalization was investigated in [14], with the goal to understand the potential of fine-tuning the dependencies between the different parameters. Dang and Doerr [14] tune all five parameters $\alpha, \gamma, \beta, A$, and $b$ using irace. Note that $\alpha$ and $\beta$ play the same roles as in Algorithm 1, while parameter $\gamma$ introduces flexibility (compared to the original $(1+(\lambda,\lambda))$-GA) in choosing $\lambda_c = \gamma\lambda_m$ for $\gamma \in \mathbb{R}$.

Based on results suggested by irace, Dang and Doerr [14] were able to derive a multi-parameter control policy that outperforms existing single-parameter control policies for the same benchmark. Their policy suggested $\alpha = 0.3594$, $\beta = 1.4128$, $\gamma = 1.2379$, $A = 1.1672$, $b = 0.691$. Their results confirm the significant performance gain when all four parameters of $(1+(\lambda,\lambda))$-GA are allowed to be chosen independently; some empirical results can be found in the subsequent sections. However, there are a number of limitations in their work. First, the derived policy required the algorithm to be slightly modified: the choices of $\lambda_m$ and $\lambda_c$ at each iteration are based on the fitness values of both the current solution and the solution obtained from the previous iteration, which may complicate theoretical analysis. Second, $\alpha$ and $\beta$ were selected statically, leaving an open question of whether there is potential gain when they are controlled dynamically.

In this work, we will take an alternative approach where we formulate the multi-parameter control task as Dynamic Algorithm Configuration (DAC) [9], where the aim is to leverage deep-RL to learn a policy that dynamically controls all four parameters $\lambda_m, \lambda_c, \alpha$, and $\beta$ without having to turn it into a static algorithm configuration problem. This would allow maximum flexibility in

## 2.3 Single-parameter Control Policies

In Algorithm 1, we allow the parameters to depend on the quality (“fitness”) of the current-best solution (cf. line 3). Our key interest is in identifying parameter settings that minimize the expected number of function evaluations that the $(1+(\lambda,\lambda))$-GA performs until it finds the optimal solution. In other terms, we seek policies that assign to each fitness value $f(x) \in \{0, \dots, n-1\}$ values for our four parameters $\lambda_m, \lambda_c, \alpha$, and $\beta$.

In the early works for this problem, only $\lambda_m$ is controlled, and a static dependence of $\lambda_c = \lambda_m$ and static choice $\alpha = \beta = 1$ was used. A key result in [19, Theorem 8] states that, with these choices and setting $\lambda_m = \sqrt{n/(n-f(x))}$, the $(1+(\lambda,\lambda))$-GA achieves a linear expected running time; a performance that is impossible to obtain with any fixed choices of $\lambda_m, \lambda_c, \alpha$, and $\beta$ [17, Section 5].

Since such a precise dependence of $\lambda_m$ and $f(x)$ is difficult to guess without considerable effort, already in [19], a self-adjusting choice of $\lambda_m$ was experimented with. More precisely, the empirical results from [19] suggest that a variant of the so-called one-fifth success rule, originally proposed in the context of evolution strategies for continuous optimization [15, 45, 47] and adapted to discrete optimization settings in [33], leads to surprisingly good performance. That it indeed leads to an expected linear running time was then formally proven in [17, Theorem 9]. For later reference in this work, we provide the details of this control policy in Algorithm 2.
**Algorithm 2:** The self-adjusting (1 + (λ, λ))-GA variant with five parameters (α, γ, β, A, b) to be tuned statically [14]. The parts that are different from Algorithm 1 are marked in blue.

**Input:** Problem size *n*, fitness function *f*(*·*), parameter values for (α, y, β, A, b)

1. *x* ← a sample from {0, 1}^*n* chosen uniformly at random;

2. $\lambda_m \leftarrow 1$;

3. **while** *x* is not the optimal solution **do**

4. **Mutation phase:**

5. $p = \alpha\lambda_m/n$;

6. Sample $\ell$ from $\text{Bin}_{>0}(n, p)$;

7. **for** *i* = 1, ..., *λ*<sub>*m*</sub> **do** *x*<sup>(*i*)</sup> ← flip<sub>*ℓ*</sub>(*x*); Evaluate *f*(*x*<sup>(*i*)</sup>);

8. Choose *x'* ∈ {*x*<sup>(1)</sup>, ..., *x*<sup>(*λ*<sub>*m*</sub>)</sup>} with
   *f*(*x'* ) = max{*f*(*x*<sup>(1)</sup>), ..., *f*(*x*<sup>(*λ*<sub>*m*</sub>)</sup>)} u.a.r.;

9. **Crossover phase:**

10. $\lambda_c = \gamma\lambda_m$;

11. $c = \beta/\lambda_c$;

12. $\mathcal{Y} \leftarrow \emptyset$;

13. **for** *i* = 1, ..., *λ*<sub>*c*</sub> **do**

14. $y^{(i)} \leftarrow \text{cross}_c(x, x')$;

15. **if** $y^{(i)} \notin \{x, x'\}$ **then** Evaluate $f(y^{(i)})$; Add $y^{(i)}$ to $\mathcal{Y}$;

16. **Selection and update step:**

17. Choose $y \in \{x'\} \cup \mathcal{Y}$ with
   $f(y) = \max\{f(x'), \max_{y' \in \mathcal{Y}} f(y')\}$ u.a.r.;

18. **if** $f(y) > f(x)$ **then**

19. $x \leftarrow y$;

20. $\lambda_m \leftarrow \max(A\lambda_m, 1)$;

21. **if** $f(y) = f(x)$ **then**

22. $x \leftarrow y$; $\lambda_m \leftarrow \max(b\lambda_m, n-1)$;

23. **if** $f(y) < f(x)$ **then**

24. $\lambda_m \leftarrow \min(b\lambda_m, n-1)$;

► mutation rate

► crossover population size

► crossover bias

► increasing $\lambda_m$

► decreasing $\lambda_m$

► decreasing $\lambda_m$

controlling those parameters. As we show in the later sections, our approach allows the derivation of a simple, yet highly effective policy that significantly outperforms any existing policies on this benchmark, including the one derived in Dang and Doerr [14].

### 3 Deep RL for Multi-parameter Control of
(1+(λ,λ))-GA on ONEMAX

Our objective is to dynamically control four parameters $\lambda_m$, $\alpha$, $\lambda_c$, $\beta$ as illustrated in Algorithm 1. Inspired by the work of Nguyen et al. [42], which demonstrates the potential of RL for single-parameter configuration on the same benchmark, we extend this line of research by formulating the multi-parameter configuration problem as an RL task [48, 49] as below.

During the execution of the algorithm, a state $s \in S$ captures all relevant information about its current status. An action $a \in \mathcal{A}$ specifies a tuple of values for the parameters to be used in the next iteration—namely, $\langle \lambda_m, \alpha, \lambda_c, \beta \rangle$. Upon selecting an action $a$

(i.e., setting parameter values), the algorithm receives a reward $r = \mathcal{R}(s, a)$ that quantifies the quality of the chosen action in the given state.

This setting naturally fits into an RL framework, where the objective is to learn a policy $\pi^*$ that maps each state $s$ to an action $a = \pi^*(s)$ in order to maximize the expected cumulative reward over time:

$$ \pi^* = \arg \max_{\pi} \mathbb{E}_{\pi} \left[ \sum_{k=0}^{\infty} \gamma^k \mathcal{R}(s_{t+k}, a_{t+k}) \mid s_t = s, a_t = a \right], \quad (1) $$

where $t$ is the current time step and $\gamma \in [0, 1)$ is the discount factor that balances immediate versus future rewards.

To deal with a multi-dimensional action space, a natural choice for the RL algorithm is Proximal Policy Optimization (PPO) [46], a versatile and well-known deep RL algorithm. Unfortunately, our preliminary experiments with PPO did not yield competitive results, even after extensive hyper-parameter optimization has been conducted. On the other hand, we were encouraged by the strong performance of the Dual Deep Q-Network (DDQN) framework [41] reported in Nguyen et al. [42] for single-parameter control on the same benchmark. Motivated by these findings, we adopt DDQN to address the more challenging multi-parameter configuration task. Briefly, DDQN is an RL algorithm that uses a neural network to approximate the action-value function $\mathcal{Q}(s, a)$, which estimates the expected return of taking action $a$ in state $s$ and following the policy thereafter.

#### 3.1 DDQN with Combinatorial Actions

When the four parameters $\langle \lambda_m, \alpha, \lambda_c, \beta \rangle$ are treated as a single atomic action, standard DDQN (as well as other RL methods) can be directly applied without requiring special treatment for the internal structure of the action space. In this work, we build on the approach proposed in Nguyen et al. [42], with extensions to support joint multi-parameter configuration.

First, to assess the difficulty of exploration in this combinatorial action space, we explore multiple reward functions proposed in Nguyen et al. [42], beginning with the conventional reward function: $r_t = -E_t + \Delta f_t$. We then apply their adaptive shifted variant $r_t = -E_t + \Delta f_t + b_a^-$, where $E_t$ is the total number of solution evaluations in the current iteration, $\Delta f_t = f(x_{t+1}) - f(x_t)$ measures the fitness improvement, and $b_a^-$ is an adaptive bias term computed from the replay buffer during the warm-up phase to stabilize learning.

Second, we investigate the influence of the discount factor $\gamma$ on learning performance. As established in prior work [30, 50, 55], the discount factor plays a critical role in reward estimation, which has been shown to be particularly sensitive in the ONEMAX benchmark [42]. A larger $\gamma$ places greater emphasis on long-term rewards, potentially encouraging more strategic behavior but at the cost of slower convergence and increased instability. Conversely, a smaller $\gamma$ prioritizes immediate rewards, facilitating faster learning but risking shortsighted policies—especially in problems with delayed or sparse rewards.

The ONEMAX problem can be modeled as an *episodic task*, where the optimization process is naturally segmented into episodes, each starting from a defined initial state and terminating either upon
<figure><img src="image_2.png" /><figcaption>Figure 1: Deep Q-network architectures for (a) combinatorial and (b) factored action space representation for (1+(λ,λ))-GA with four controllable parameters, each of which is selected from a set of 7 possible values. The number of output nodes for the combinatorial representation is 7⁴ = 2,401.</figcaption></figure>

reaching the global optimum or hitting a predefined time cutoff. As recommended in Sutton and Barto [50], high or even undiscounted reward estimations are preferable in episodic tasks to avoid underestimating future gains. Accordingly, we evaluate two values for the discount factor: $γ \in \{0.99, 0.9998\}$, with the latter providing minimal discounting to better capture long-horizon reward signals.

Third, we modify the Q-network architecture proposed in Nguyen et al. [42] (illustrated in Figure 1a) such that its output layer spans the full combinatorial action space. At inference time, given a state s, the agent selects the optimal action according to:

$$a = \arg\max_{\langle\lambda_m, \alpha, \lambda_c, \beta\rangle} Q(s, \langle\lambda_m, \alpha, \lambda_c, \beta\rangle) \quad (2)$$

## 3.2 DDQN with Factored Actions

Treating each unique combination of sub-actions (i.e., individual parameter values) as a distinct atomic action—without considering the internal structure—is inefficient, especially when each sub-action can take on a large number of possible values. Specifically, if there are $n_d$ possible values for each sub-action and an action is composed of $D$ sub-actions $\langle a_1, \dots, a_D \rangle$, then the total action space grows exponentially as $\prod_{d=1}^D n_d$. Such a combinatorial explosion poses serious challenges for RL, particularly for value-based methods like DDQN, as it hinders learning efficiency and increases inference cost [13, 23, 24, 38]. To address this issue, prior work has explored hierarchical action selection [28], common-sense constraints [44], and learning-based reduction strategies [59] to effectively prune the action space.

In our second RL approach, we leverage the structure of factored actions by decomposing each action into $D$ sub-actions, following methods from e.g., Tavakoli et al. [51]. Each sub-action is modeled independently using a dedicated architecture. Concretely, for each dimension $d$, we define a distinct action-value function $Q_d(s, a_d)$ that estimates the expected return of taking sub-action $a_d$ in state s, according to a local policy $\pi_d$. These functions are trained independently and capture only the marginal contribution of each dimension, without explicitly modeling interactions or dependencies between sub-actions.

This decomposition dramatically reduces the effective action space size and enhances scalability. To operationalize this structure,

we adopt the action branching framework introduced by Tavakoli et al. [51], which offers an efficient and empirically validated approach for learning in factored action spaces. In our implementation, we construct a shared neural network backbone (illustrated in Figure 1b) with $D = 4$ output branches (or “heads”), each corresponding to a sub-action-specific $Q_d$ function. At inference time, the final joint action $a$ is constructed by independently selecting the optimal value in each sub-action dimension:

$$a = \begin{cases} \arg\max_{\lambda_m} Q_1(s, \lambda_m) \\ \arg\max_{\alpha} Q_2(s, \alpha) \\ \arg\max_{\lambda_c} Q_3(s, \lambda_c) \\ \arg\max_{\beta} Q_4(s, \beta) \end{cases} \quad (3)$$

## 3.3 Experimental Setup

Because DDQN requires action spaces to be discrete, we first need to discretize the domains of (1+(λ,λ))-GA’s parameters. Following Nguyen et al. [42], we define the domain of $\lambda_m$ and $\lambda_c$ as $D_1 = \{1, 2, 4, 8, 16, 32, 64\}$. The upper bound was chosen based on results suggested in Nguyen et al. [42], where it was observed that the population size in the best policies learned by deep-RL never exceeded 64 for all tested problem sizes of $n \le 2000$. For $\alpha$ and $\beta$, we use [0.25, 2] as the domain following the setting in Dang and Doerr [14]. We discretize this domain uniformly for our DDQN experiments. For simplicity, we use the same number of values as in $D_1$, i.e., $\alpha$ and $\beta$ are chosen from the set of $D_2 = \{0.25, 0.542, 0.833, 1.125, 1.417, 1.708, 2\}$.

Each deep Q-network has 2 hidden layers, each with 50 nodes. For the combinatorial representation, the output layer is comprised of $7^4 = 2,401$ output nodes.

In line with Biedenkapp et al. [10], Nguyen et al. [42], we use a standard DDQN hyperparameter setting with $\epsilon$-greedy exploration strategy where $\epsilon = 0.2$. The replay buffer is configured to store 1 million transitions. Prior to training, we initialize the replay buffer by randomly sampling 10,000 transitions. For optimization, we employ the Adam optimizer [35] with a (mini-)batch size of 2,048 and a learning rate of 0.001. A cutoff time of $0.8n^2$ solution evaluations, where $n$ is the problem size, is imposed on each (1+(λ,λ))-GA run to avoid wasting training budget on bad policies.

RL training is conducted per problem size with a training budget of 200,000 training time steps for each RL training run. Following
previous work Nguyen et al. [42] on the same benchmark, to select the best policy from each RL training, we evaluate the learned policy at every 2,000 training time steps with a budget of 100 runs per evaluation. At the end of the training, we collect the top 5 best learned policies, re-evaluate each of them 1,000 times and extract the best one. As deep-RL is commonly known to be unstable [29, 31], and since our main focus in this work is to leverage deep-RL to discover new parameter control policies for this particular benchmark, for each RL setting, we repeat the training 5 times and use the best policy among those trainings as the final policy.

All experiments are conducted on a machine equipped with a single-socket AMD EPYC 7443 24-Core Processor. For the RL training phase we use a single core, while for the evaluation phase, we parallelize 16 threads. The total wall-time for each RL training ranges between 30 minutes to one hour, depending on the problem size. This budget is smaller than the irace-based approach where a tuning was reported to take up to 2 days with 20 parallel threads [14].

**Performance metric.** In the rest of this paper, the performance of each parameter control policy is reported using the Expected Runtime (ERT) across 1000 random seeds. The runtime here is defined as the total number of solution evaluations used during a run of the $(1+(\lambda,\lambda))$-GA.

**Statistical tests.** In all result tables, the policy with the best ERT is highlighted in **bold**. We perform a Wilcoxon signed-rank test (confidence level 0.99) comparing each of the remaining policies against the best one. Policies that are not statistically significantly different from the best are **underlined**. We also apply the Holm-Bonferroni method to correct for multiple comparisons.

## 3.4 Results

We first evaluate the impact of the three DDQN's design choices described in Section 3.1 and Section 3.2, including: (i) action space representation (combinatorial vs. factored); (ii) reward function (naive vs. adaptive reward shifting, as in Nguyen et al. [42]); and (iii) discount factor ($\gamma \in \{0.99, 0.9998\}$).

Table 1 presents the performance of all combinations of those three design choices on three problem sizes $n \in \{100, 200, 500\}$. Interestingly, when the naive reward function and the commonly used discount factor $\gamma = 0.99$ are used, there is no clear advantage in using factored action space representation. However, when adaptive reward shifting is adopted, the factored representation starts outperforming its combinatorial counterpart on 2 (/3) problem sizes. When the discount factor is also adjusted to reflect the long episode-horizon characteristic of the benchmark (the last row in Table 1), the factored representation shows a clear overall advantage over the combinatorial one: for the problem sizes of $n = 100$ and $n = 500$, the learned policies are statistically significantly better than all the rest, while for $n = 200$, it is not statistically significantly different from the best combination.

Our results not only confirm the advantage of the factored action space representation, but also highlight the strong interaction between this design choice and the other two components (reward shifting and discount factor adjustment). This observation indicates the intricacy of applying deep-RL in DAC contexts, underscoring

<figure><img src="image_3.png" /><figcaption>**Figure 2:** Ablation path from controlling only one parameter of $(1+(\lambda,\lambda))$-GA on ONEMax to controlling all four parameters with DDQN across different problem sizes. Each point shows the normalized ERTs over 1,000 independent runs and the standard deviation of the corresponding learned policy.</figcaption></figure>

the importance of jointly optimizing architectural and algorithmic choices when designing effective deep RL solutions for DAC.

Having established that the combination of factored representation, adaptive reward shifting, and discount factor adjustment is the best overall setting for DDQN, we conduct a new set of experiments on a larger set of problem sizes $n \in \{100, 200, 500, 1000, 1500, 2000\}$ using the chosen DDQN setting. In Table 2, we compare the learned policies obtained from our DDQN-based approach, denoted as $\pi_{MP}$, with several baselines from the literature as described in Section 2.3 and Section 2.4, including:

* the theory-derived single-parameter control policy proposed in [17, 19], namely $\pi_{THEORY}$;

* the single-parameter control policy based on the one-fifth success rule [7, 34];

* the multi-parameter control policy obtained from IRACE [14];

* the DDQN-based single-parameter control policies obtained in [42], namely $\pi_{\lambda}$.³

Note that we do not explicitly compare to the results presented in [3] and [4] since both of them are much worse than those obtained by $\pi_{THEORY}$, with ERTs exceeding $12n$ for [3] and results worse than the $(1+1)$ EA (and hence much worse than $\pi_{THEORY}$) in [4].

Among the baselines, the IRACE-based approach [14] is the strongest one. This is not too surprising, as this is the only baseline that tunes all four parameters of $(1+(\lambda,\lambda))$-GA separately. Nevertheless, our multi-parameter control DDQN-based approach statistically significantly outperforms all baselines, including the IRACE-based one, across all tested problem sizes.

Our results strongly confirm the advantage of multi-parameter control of $(1+(\lambda,\lambda))$-GA on ONEMax. In the subsequent sections, we will leverage the obtained results to derive a simple yet powerful control policy for this benchmark.
<table><caption>Table 1: ERT (and its standard deviation) normalized by the problem size *n* across 1000 seeds of the best policies achieved by each combination of Combinatorial and Factored action representations (AR), Naïve and Adaptive Shifting (AS) reward functions, and discount factors ($\gamma$), across three problem sizes $n \in \{100, 200, 500\}$.</caption><thead><tr><th>AR</th><th>Reward</th><th>γ</th><th>n = 100</th><th>n = 200</th><th>n = 500</th></tr></thead><tbody><tr><td>C</td><td>Naïve</td><td>0.99</td><td>4.510<sub>(1.07)</sub></td><td>5.057<sub>(0.83)</sub></td><td>5.959<sub>(0.79)</sub></td></tr><tr><td>C</td><td>AS</td><td>0.99</td><td>4.392<sub>(0.77)</sub></td><td>4.961<sub>(0.76)</sub></td><td>5.583<sub>(0.74)</sub></td></tr><tr><td>C</td><td>Naïve</td><td>0.9998</td><td>4.350<sub>(0.80)</sub></td><td>4.969<sub>(0.72)</sub></td><td>6.531<sub>(0.61)</sub></td></tr><tr><td>C</td><td>AS</td><td>0.9998</td><td>4.777<sub>(0.81)</sub></td><td>4.835<sub>(0.59)</sub></td><td>5.198<sub>(0.51)</sub></td></tr><tr><td>F</td><td>Naïve</td><td>0.99</td><td>4.952<sub>(1.38)</sub></td><td>5.731<sub>(1.42)</sub></td><td>6.835<sub>(1.47)</sub></td></tr><tr><td>F</td><td>AS</td><td>0.99</td><td>4.395<sub>(0.97)</sub></td><td>4.689<sub>(0.79)</sub></td><td>5.948<sub>(0.93)</sub></td></tr><tr><td>F</td><td>Naïve</td><td>0.9998</td><td>4.467<sub>(0.98)</sub></td><td><strong>4.472</strong><sub>(0.63)</sub></td><td>4.932<sub>(0.52)</sub></td></tr><tr><td>F</td><td>AS</td><td>0.9998</td><td><strong>4.318</strong><sub>(0.83)</sub></td><td><u>4.483</u><sub>(0.65)</sub></td><td><strong>4.814</strong><sub>(0.42)</sub></td></tr></tbody></table>

<table><caption>Table 2: ERT (and its standard deviation) normalized by the problem size *n* across 1000 seeds of the best policies found by our DDQN-based multi-parameter control approach and of several baselines from the literature.</caption><thead><tr><th></th><th>π<sub>THEORY</sub> [17, 19]</th><th>IRACE [14]</th><th>1/5-th [19]</th><th>π<sub>λ</sub> [42]</th><th>π<sub>MP</sub></th></tr></thead><tbody><tr><td>n = 100</td><td>5.826<sub>(1.18)</sub></td><td>4.990<sub>(0.86)</sub></td><td>6.197<sub>(1.29)</sub></td><td>5.442<sub>(1.23)</sub></td><td><strong>4.318</strong><sub>(0.83)</sub></td></tr><tr><td>n = 200</td><td>6.167<sub>(0.97)</sub></td><td>5.216<sub>(0.69)</sub></td><td>6.541<sub>(1.09)</sub></td><td>5.711<sub>(0.96)</sub></td><td><strong>4.483</strong><sub>(0.65)</sub></td></tr><tr><td>n = 500</td><td>6.474<sub>(0.67)</sub></td><td>5.478<sub>(0.46)</sub></td><td>6.744<sub>(0.70)</sub></td><td>6.017<sub>(0.63)</sub></td><td><strong>4.814</strong><sub>(0.42)</sub></td></tr><tr><td>n = 1000</td><td>6.587<sub>(0.53)</sub></td><td>5.587<sub>(0.35)</sub></td><td>6.886<sub>(0.54)</sub></td><td>6.338<sub>(0.55)</sub></td><td><strong>5.397</strong><sub>(0.41)</sub></td></tr><tr><td>n = 1500</td><td>6.647<sub>(0.44)</sub></td><td>5.621<sub>(0.29)</sub></td><td>6.931<sub>(0.48)</sub></td><td>6.209<sub>(0.41)</sub></td><td><strong>4.971</strong><sub>(0.29)</sub></td></tr><tr><td>n = 2000</td><td>6.681<sub>(0.39)</sub></td><td>5.666<sub>(0.26)</sub></td><td>7.008<sub>(0.41)</sub></td><td>6.608<sub>(0.41)</sub></td><td><strong>5.162</strong><sub>(0.29)</sub></td></tr></tbody></table>

<table><caption>Table 3: ERT (and its standard deviation) normalized by the problem size *n* across 1000 seeds of different versions with varying numbers of controlled parameters. The term RL denotes dynamic control achieved through reinforcement learning, whereas $\alpha = \beta = 1$ and $\lambda_c = \lambda_m$ indicate no dynamic control.</caption><thead><tr><th>λ<sub>m</sub></th><th>α</th><th>λ<sub>c</sub></th><th>β</th><th>n = 500</th><th>n = 1000</th><th>n = 1500</th><th>n = 2000</th></tr></thead><tbody><tr><td colspan="8">π<sub>THEORY</sub> [17, 19]</td></tr><tr><td>RL</td><td>1</td><td>λ<sub>m</sub></td><td>1</td><td>6.017<sub>(0.63)</sub></td><td>6.338<sub>(0.55)</sub></td><td>6.209<sub>(0.41)</sub></td><td>6.608<sub>(0.41)</sub></td></tr><tr><td>RL</td><td>1</td><td>RL</td><td>1</td><td>5.052<sub>(0.48)</sub></td><td>5.268<sub>(0.38)</sub></td><td>5.504<sub>(0.39)</sub></td><td>6.133<sub>(0.39)</sub></td></tr><tr><td>RL</td><td>RL</td><td>λ<sub>m</sub></td><td>1</td><td>5.064<sub>(0.60)</sub></td><td>5.187<sub>(0.37)</sub></td><td>5.282<sub>(0.39)</sub></td><td>5.277<sub>(0.34)</sub></td></tr><tr><td>RL</td><td>1</td><td>λ<sub>m</sub></td><td>RL</td><td>6.170<sub>(0.67)</sub></td><td>6.220<sub>(0.53)</sub></td><td>6.384<sub>(0.44)</sub></td><td>6.722<sub>(0.44)</sub></td></tr><tr><td>RL</td><td>RL</td><td>λ<sub>m</sub></td><td>RL</td><td>5.191<sub>(0.63)</sub></td><td>5.303<sub>(0.43)</sub></td><td>5.609<sub>(0.33)</sub></td><td>5.410<sub>(0.32)</sub></td></tr><tr><td>RL</td><td>RL</td><td>RL</td><td>1</td><td>4.738<sub>(0.40)</sub></td><td><strong>4.869</strong><sub>(0.34)</sub></td><td>5.196<sub>(0.34)</sub></td><td>5.297<sub>(0.32)</sub></td></tr><tr><td>RL</td><td>RL</td><td>RL</td><td>RL</td><td><strong>4.814</strong><sub>(0.42)</sub></td><td><u>5.397</u><sub>(0.41)</sub></td><td><strong>4.971</strong><sub>(0.29)</sub></td><td><strong>5.162</strong><sub>(0.29)</sub></td></tr></tbody></table>

## 4 Ablation Study

Since our proposed method outperforms others by controlling multiple parameters, we conduct ablation analyses to identify the significance of each parameter. We train the DDQN model with the best setting – using the factored action space, adaptive reward shifting, and setting the discount factor $\gamma = 0.9998$ – on four problem sizes: {500, 1000, 1500, 2000}.

The ablation starts with training DDQN to control only one parameter, which is the mutation population size ($\lambda_m$), while the remaining parameters ($\alpha, \lambda_c, \beta$) are determined by the theory-derived policy $\pi_{THEORY}$, i.e., $\lambda_c = \lambda_m$ and $\alpha = \beta = 1$. Gradually, we integrate

more parameters into the set of controllable parameters by RL. For instance, in the second iteration of the ablation, we conduct RL trainings for three combinations: ($\lambda_m, \lambda_c$), ($\lambda_m, \alpha$), and ($\lambda_m, \beta$). We then select the most effective combination as the starting point for the third step, and so on. This is repeated until all four parameters are included.

Table 3 shows the performance of all combinations in each ablation iteration across the four studied problem sizes.

A summary version of Table 3 where only the best combination at each iteration is shown is presented in Figure 2.

When comparing the theory-derived policy and the policies obtained from simultaneously controlling mutation population size

<footnote n="3">³The ERT reported in [42] was based on average performance across 5 RL training runs. To ensure a fair comparison, we re-run their experiments and report performance of the best policy across 5 RL training runs instead of taking the average.</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 3: DDQN-based policies and the theory-derived policy across six problem sizes of $n \in \{100, 200, 500, 1000, 1500, 2000\}$. Note that the evolution path of $\lambda_m$ and $\lambda_c$ in the theory-derived policy $\pi_{\text{THEORY}}$ is exactly the same for all problem sizes, however, the upper bound for those parameters increases as problem size increases. Here we plot those values to the maximum upper bound, which corresponds to $n = 2000$. In addition to showing the four parameters being controlled by RL, we also show the corresponding mutation rate $p$ and crossover bias $c$ values (third column).</figcaption></figure>

($\lambda_m$) and the mutation rate coefficient ($\alpha$), the average improvements achieved are over 18%. Although the average expected runtimes of ($\lambda_m, \alpha$) do not always outperform ($\lambda_m, \lambda_c$), our Wilcoxon signed-rank test's results suggest that there is no statistical difference between these two combinations. This observation highlights the crucial role of $\alpha$ in almost four tested problem sizes, and we conclude that $\alpha$ is the next parameter that should be controlled. Therefore, we then verify the impact of integrating one of the two parameters of the crossover phase into the combination. The advancements increase over 22% when we incorporate the offspring population size of crossover ($\lambda_c$). On the contrary, replacing $\lambda_c$ with $\beta$ adversely affects the overall performance. Controlling ($\lambda_m, \beta$) also leads to the same observations, suggesting that dynamically adjusting the factor of crossover bias ($\beta$) is not an effective way to enhance the optimization process.

The common trend across four problem sizes is shown in Figure 2, where we see an overall improvement in most cases when a new parameter is added into the set of controllable parameters. There are only a few exceptions, such as the last point in the ablation path of $n = 1000$ where the performance got worse. We attribute those cases to DDQN's learning instability and the challenges when the size of the action space is increased. This observation is not too surprising as this kind of learning instability behavior has been observed in previous work on theory-derived dynamic algorithm configuration benchmarks [10, 42]. We leave for future work a more thorough investigation into improving the learning stability in such cases. In overall, we observe intense plummets during the transition from controlling $\lambda_m$ alone to ($\lambda_m, \alpha$), followed by ($\lambda_m, \alpha, \lambda_c$), and finally,

we achieve full control over four parameters. This order reflects the important degree of each parameter: $\lambda_m \gg \alpha \gg \lambda_c \gg \beta$.

# 5 Derived Multi-parameter Control Policy

Although the policies learned via RL shows strong performance, they are problem-dependent. Moreover, as a neural network-based approach, each learned policy functions as a black box. Those aspects may hinder the usefulness of our findings for theoretical insights. To address this limitation, in this section, based on our ablation analysis and the policies suggested by DDQN, we derive a new symbolic policy, namely $\pi_{\text{DMP}}$, that is not only highly interpretable but also exhibits strong performance and generalization. As shown later in this section, our newly derived policy consistently outperforms all existing baselines even on very large problem sizes. We hope this new policy serves as a foundation for future theoretical work on multi-parameter control for $(1+(\lambda,\lambda))$-GA on ONEMAX.

**Definition and Justification for our new policy $\pi_{\text{DMP}}$.** To derive a new policy, we visualize the DDQN-based policies across all problem sizes based on the normalized fitness values, together with the theory-derived policy, in Figure 3. We first look at $\lambda_m$, an important parameter as indicated in our ablation study in Section 4. In general, this parameter is set as one by DDQN for most of the fitness range. Only later in the search, where the fitness value is around 0.95$n$ on average, that $\lambda_m$ starts increasing. The magnitudes of $\lambda_m$ in that later part is fairly close to the values suggested by theory (i.e., $\lambda_m = \sqrt{n/(n-f(x))}$).
<figure><img src="image_5.png" /><figcaption>Figure 4: ERT (and its standard deviation) normalized by the problem size *n* across 1000 seeds of our proposed derived policy $\pi_{DMP}$, compared to the theory-derived policy [17, 19] and the IRACE-based approach in [14].</figcaption></figure>

<table><caption>Table 4: ERT (and its standard deviation) normalized by the problem size *n* across 1000 seeds for seven problem sizes of different versions of the derived multi-parameter control policy. The final combination of $\pi_{DMP}$ is represented by the last row. Each term in Eq. (·) corresponds to the equation of the new symbolic form for each parameter, while the reference to [17, 19] refers to the theory formula $\lambda_m = \sqrt{n/(n - f(x))}$. The cases where $\alpha = \beta = 1$ and $\lambda_c = \lambda_m$ denote no dynamic control.</caption><thead><tr><th>λ<sub>m</sub></th><th>α</th><th>λ<sub>c</sub></th><th>β</th><th>n = 3,000</th><th>n = 4,000</th><th>n = 5,000</th><th>n = 10,000</th><th>n = 20,000</th><th>n = 30,000</th><th>n = 40,000</th></tr></thead><tbody><tr><td colspan="11">π<sub>THEORY</sub> [17, 19]</td></tr><tr><td>Eq.(4)</td><td>1</td><td>λ<sub>m</sub></td><td>1</td><td>6.723<sub>(0.32)</sub></td><td>6.763<sub>(0.28)</sub></td><td>6.768<sub>(0.26)</sub></td><td>6.834<sub>(0.20)</sub></td><td>6.855<sub>(0.14)</sub></td><td>6.877<sub>(0.13)</sub></td><td>6.882<sub>(0.11)</sub></td></tr><tr><td>[17, 19]</td><td>Eq.(6)</td><td>λ<sub>m</sub></td><td>1</td><td>6.353<sub>(0.35)</sub></td><td>6.384<sub>(0.30)</sub></td><td>6.404<sub>(0.28)</sub></td><td>6.475<sub>(0.21)</sub></td><td>6.545<sub>(0.16)</sub></td><td>6.581<sub>(0.15)</sub></td><td>6.602<sub>(0.14)</sub></td></tr><tr><td>[17, 19]</td><td>1</td><td>Eq.(5)</td><td>1</td><td>5.905<sub>(0.32)</sub></td><td>5.957<sub>(0.30)</sub></td><td>5.980<sub>(0.26)</sub></td><td>6.044<sub>(0.20)</sub></td><td>6.096<sub>(0.16)</sub></td><td>6.142<sub>(0.15)</sub></td><td>6.157<sub>(0.14)</sub></td></tr><tr><td></td><td></td><td></td><td></td><td>5.891<sub>(0.24)</sub></td><td>5.923<sub>(0.22)</sub></td><td>5.930<sub>(0.20)</sub></td><td>5.978<sub>(0.15)</sub></td><td>6.029<sub>(0.12)</sub></td><td>6.051<sub>(0.11)</sub></td><td>6.076<sub>(0.10)</sub></td></tr><tr><td>[17, 19]</td><td>Eq.(6)</td><td>Eq.(5)</td><td>1</td><td>5.167<sub>(0.24)</sub></td><td>5.199<sub>(0.21)</sub></td><td>5.216<sub>(0.19)</sub></td><td>5.269<sub>(0.14)</sub></td><td>5.309<sub>(0.12)</sub></td><td>5.336<sub>(0.11)</sub></td><td>5.362<sub>(0.10)</sub></td></tr><tr><td>Eq.(4)</td><td>1</td><td>Eq.(5)</td><td>1</td><td>6.006<sub>(0.26)</sub></td><td>6.022<sub>(0.23)</sub></td><td>6.045<sub>(0.20)</sub></td><td>6.090<sub>(0.16)</sub></td><td>6.132<sub>(0.13)</sub></td><td>6.156<sub>(0.11)</sub></td><td>6.172<sub>(0.10)</sub></td></tr><tr><td>Eq.(4)</td><td>Eq.(6)</td><td>Eq.(5)</td><td>1</td><td><strong>4.827</strong><sub>(0.23)</sub></td><td><strong>4.859</strong><sub>(0.21)</sub></td><td><strong>4.870</strong><sub>(0.19)</sub></td><td><strong>4.930</strong><sub>(0.15)</sub></td><td><strong>4.964</strong><sub>(0.12)</sub></td><td><strong>4.998</strong><sub>(0.12)</sub></td><td><strong>5.006</strong><sub>(0.10)</sub></td></tr></tbody></table>

The next most important parameter according to the ablation is $\alpha$, the mutation rate coefficient. A common trend for this parameter is that it starts jumping to a high value when $f(x) \ge 0.95n$. Therefore, we design a policy where $\alpha$ is set as a small value (0.001) during the first part of the search,<fnref n="4" />, and is then increased to 1 when $f(x)$ reaches 0.95n.

The third parameter we look at is $\lambda_c$, the population size for the crossover phase. Notably, although the theory-derived policy suggests $\lambda_m = \lambda_c$, our DDQN approach imposes a much larger value for $\lambda_c$. The range of this parameter is approximately twice the one suggested by theory. This is similar to the findings in [14], where IRACE suggested that $\lambda_c = 2\lambda_m$. Our DDQN-based approach is able to recognize this correlation without having to explicitly formulate the relationship between $\lambda_c$ and $\lambda_m$. This allows for more flexibility compared to the IRACE-based approach: since $\lambda_m$ and $\lambda_c$ are completely decoupled from each other, DDQN can choose to

“delay” increasing $\lambda_m$ during the early part of the search. As we will show in the ablation analysis at the end of this section (Table 4), this design choice has a clear positive contribution to the performance difference between our derived policy and the one found by IRACE.

Finally, we look at $\beta$, the crossover bias coefficient. The values learned by DDQN for this parameter show a less clear trend, and since this parameter was indicated in our ablation in Section 4 as the least important one among the four parameters, we decide to keep this parameter as a constant of 1, i.e., similar to the theory-derived policy.

In summary, we derive a new multi-parameter control policy $\pi_{DMP}$ as follows:

* Population size for the mutation phase:

$$ \lambda_m = \begin{cases} 1 & \text{if } f(x)/n \le 0.95 \\ \sqrt{n/(n - f(x))} & \text{otherwise} \end{cases} \qquad (4) $$

* Population size for the crossover phase:

$$ \lambda_c = 2\sqrt{n/(n - f(x))} \qquad (5) $$

<footnote n="4">⁴Note that in Figure 3 for a few problem sizes, DDQN appears to suggest a “U-shape” policy where $\alpha$ receives a large value before dropping and then increasing. We also experimented with this U-shape policy where we set $\alpha = 0.5$ if $f(x) < 0.85n$, $\alpha = 0.001$ if $0.85n \le f(x) \le 0.95n$, and $\alpha = 1$ otherwise. However, this policy performs worse than the one where we simply set $\alpha = 0.001$ for all $f(x) \le 0.95n$. The differences are small, though, this alternative U-shaped policy achieves a normalized ERT of around 6.212 averaged across 7 tested problem sizes.</footnote>
* Mutation rate coefficient:

$$ \alpha = \begin{cases} 0.001 & \text{if } f(x)/n \le 0.95 \\ 1 & \text{otherwise} \end{cases} \qquad (6) $$

* Crossover bias coefficient: $\beta = 1$

**Empirical evaluation of $\pi_{DMP}$**. We evaluate the newly derived policy across a range of problem sizes from $n = 500$ to $n = 40,000$ and compare against both the theory-derived policy and the multi-parameter control policy found by IRACE [14] (the strongest baseline on this benchmark). As shown in Figure 4, our policy significantly outperforms the others across all tested problem sizes. Interestingly, the normalized ERT appears to be constant across all problem sizes: it is roughly 6.9, 5.8 and 5.0 for theory-derived policy, the IRACE-based one, and our DDQN-derived policy, respectively.

**Ablation study for $\pi_{DMP}$**. To confirm the importance of each component of the newly derived policy, we conduct another ablation where we replace each of the three parameters, $\lambda_m$, $\alpha$, and $\lambda_c$, with the theory-derived policy. The results are shown in Table 4, where the expected runtime values are again normalized by the problem size.

We see that $\lambda_c$ plays a crucial role in enhancing ERT, resulting in an average improvement of over 12%, followed by the mutation rate coefficient $\alpha(\approx 11.5\%)$. We next examine the combination of two parameters, where the set of ($\alpha, \lambda_c$) exhibits an average improvement of over 22.5%, significantly outperforming the set of ($\lambda_m, \lambda_c$) (which only yields a 10% improvement). This observation highlights the negative impact of merging $\lambda_m$ into $\lambda_c$. On the other hand, $\pi_{DMP}$ that includes a tuple of ($\lambda_m, \alpha, \lambda_c$) precisely captures the contribution of each designed factor. The gap percentage is approximately the sum of each individual gap. Finally, we see that every single parameter in the ablation path has a positive impact on the performance. This observation confirms the importance of all three design choices for $\lambda_m$, $\alpha$, and $\lambda_c$ in our newly derived policy, which outperforms $\pi_{THEORY}$ by 27.5%.

# 6 Conclusion

In this work, we have shown that (deep) RL can be a powerful tool to discover high-performing multi-parameter control policies for the $(1+(\lambda, \lambda))-GA$ on ONEMAX. Based on results suggested by deep RL, we are able to derive a parameter control policy that outperforms all existing policies on this task.

As mentioned above, we believe our work to provide new insight for researchers with an interest in reinforcement learning or algorithm configuration and for researchers interested in theoretical aspects of evolutionary computation. For the latter, it is worth noting that our derived policy $\pi_{DMP}$ is fairly simple and may be tractable for a fine-grained running time analysis. However, we note that, to date, no reasonably good bounds for the constant factors of the linear expected running time of the $(1+(\lambda, \lambda))-GA$ with suitable dynamic parameter choices are known, even though experimental results for up to $n = 2^{22}$ in Antipov et al. [3] suggest that the constant factor of the default 1/5-th success rule proposed in [17, 19] is somewhere around 6.9, perfectly in line with our empirical results. The interested may have noticed that the constant factors for all algorithms and parameter control policies increase

with increasing problem dimension $n$, and this includes the settings for which we have proven guarantees of linear running times. However, as Antipov et al. [3, Section 4.3] acknowledges, the few formally proven upper bounds for the constant factors (for their algorithm choosing $\lambda$ from a heavy-tailed power-law distribution) seem to be far too pessimistic. Since the precision of running time results has drastically increased over the last years, fine-grained performance analyses of the original self-adjusting $(1+(\lambda, \lambda))-GA$ and the $(1+(\lambda, \lambda))-GA$ equipped with the $\pi_{DMP}$ policy may already be feasible or become feasible in the near future.

From a more general perspective, we hope that this work, in line with other recent works such as [54] (expanding upon [37]) and [20] (deriving running time guarantees for observations made in [22]) underlines the benefits of tying sound empirical investigations with rigorous theoretical analyses, and vice-versa.

On the empirical side, in addition to further exploring the DDQN approach used in this work in the context of dynamic algorithm configuration, we plan to conduct an extensive and systematic empirical study into other (deep) RL approaches for multi-parameter control, such as multi-agent deep RL [56]. Additionally, we aim to explore the potential performance improvements that could be achieved by learning in continuous action spaces. This would enable more fine-grained control over the four parameters but would necessitate adopting a different class of deep RL algorithms, such as Proximal Policy Optimization (PPO) [46] or Soft Actor-Critic (SAC) [27]. A further promising direction for future research involves leveraging symbolic regression techniques to automatically derive interpretable parameter control policies, guided by the insights obtained from deep RL models.

# Acknowledgments

The project is financially supported by the European Union (ERC, "dynaBBO", grant no. 101125586), by ANR project ANR-23-CE23-0035 Opt4DAC, and by an International Emerging Action funded by CNRS Sciences informatiques. This work used the supercomputer at MeSU Platform (https://sacado.sorbonne-universite.fr/plateformemesu). Tai Nguyen acknowledges funding from the St Andrews Global Doctoral Scholarship programme. This publication is based upon work from COST Action CA22137 "Randomized Optimization Algorithms Research Network" (ROAR-NET), supported by COST (European Cooperation in Science and Technology).

# References

[1] Steven Adriaensen, André Biedenkapp, Gresa Shala, Noor Awad, Theresa Eimer, Marius Lindauer, and Frank Hutter. 2022. Automated dynamic algorithm configuration. Journal of Artificial Intelligence Research 75 (2022), 1633–1699.

[2] Aldeida Aleti and Irene Moser. 2016. A Systematic Literature Review of Adaptive Parameter Control Methods for Evolutionary Algorithms. Comput. Surveys 49 (2016), 56:1–56:35.

[3] Denis Antipov, Maxim Buzdalov, and Benjamin Doerr. 2022. Fast Mutation in Crossover-Based Algorithms. Algorithmica 84, 6 (2022), 1724–1761. doi:10.1007/S00453-022-00957-5

[4] Denis Antipov, Maxim Buzdalov, and Benjamin Doerr. 2024. Lazy Parameter Tuning and Control: Choosing All Parameters Randomly from a Power-Law Distribution. Algorithmica 86, 2 (2024), 442–484. doi:10.1007/S00453-023-01098-Z

[5] Denis Antipov, Benjamin Doerr, and Vitalii Karavaev. 2019. A tight runtime analysis for the $(1 + (\lambda, \lambda))$ GA on LeadingOnes. In Proc. of ACM/SIGEVO Conference on Foundations of Genetic Algorithms (FOGA), Tobias Friedrich, Carola Doerr, and Dirk V. Arnold (Eds.). ACM, 169–182. doi:10.1145/3299904.3340317

[6] Denis Antipov, Benjamin Doerr, and Vitalii Karavaev. 2022. A Rigorous Runtime Analysis of the $(1 + (\lambda, \lambda))$ GA on Jump Functions. Algorithmica 84, 6 (2022),
1573-1602. doi:10.1007/S00453-021-00907-7

[7] Anne Auger. 2009. Benchmarking the (1+1) evolution strategy with one-fifth success rule on the BBOB-2009 function testbed. In Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers. 2447-2452.

[8] Anton O. Bassin, Maxim V. Buzdalov, and Anatoly A. Shalyto. 2021. The "One-Fifth Rule" with Rollbacks for Self-Adjustment of the Population Size in the (1 + (λ, λ)) Genetic Algorithm. Autom. Control. Comput. Sci. 55, 7 (2021), 885-902. doi:10.3103/S0146411621070208

[9] André Biedenkapp, H Furkan Bozkurt, Theresa Eimer, Frank Hutter, and Marius Lindauer. 2020. Dynamic algorithm configuration: Foundation of a new meta-algorithmic framework. In ECAI 2020. IOS Press, 427-434.

[10] André Biedenkapp, Nguyen Dang, Martin S Krejca, Frank Hutter, and Carola Doerr. 2022. Theory-inspired parameter control benchmarks for dynamic algorithm configuration. In Proceedings of the Genetic and Evolutionary Computation Conference. 766-775.

[11] Maxim Buzdalov and Benjamin Doerr. 2017. Runtime analysis of the $(1 + (\lambda, \lambda))$ genetic algorithm on random satisfiable 3-CNF formulas. In Proc. of Genetic and Evolutionary Computation Conference (GECCO), Peter A. N. Bosman (Ed.). ACM, 1343-1350. doi:10.1145/3071178.3071297

[12] Deyao Chen, Maxim Buzdalov, Carola Doerr, and Nguyen Dang. 2023. Using automated algorithm configuration for parameter control. In Proceedings of the 17th ACM/SIGEVO Conference on Foundations of Genetic Algorithms. 38-49.

[13] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep neural networks for youtube recommendations. In Proceedings of the 10th ACM conference on recommender systems. 191-198.

[14] Nguyen Dang and Carola Doerr. 2019. Hyper-parameter tuning for the $(1+(\lambda, \lambda))$ GA. In Proceedings of the Genetic and Evolutionary Computation Conference. 889-897.

[15] Luc Devroye. 1972. The compound random search. Ph.D. dissertation, Purdue Univ., West Lafayette, IN.

[16] Benjamin Doerr and Carola Doerr. 2015. Optimal parameter choices through self-adjustment: Applying the 1/5-th rule in discrete settings. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation. 1335-1342.

[17] Benjamin Doerr and Carola Doerr. 2018. Optimal static and self-adjusting parameter choices for the $(1+(\lambda, \lambda))$ genetic algorithm. Algorithmica 80 (2018), 1658-1709.

[18] Benjamin Doerr and Carola Doerr. 2020. Theory of parameter control for discrete black-box optimization: Provable performance gains through dynamic parameter choices. Theory of Evolutionary Computation: Recent Developments in Discrete Optimization (2020), 271-321.

[19] Benjamin Doerr, Carola Doerr, and Franziska Ebel. 2015. From black-box complexity to designing new genetic algorithms. Theoretical Computer Science 567 (2015), 87-104.

[20] Benjamin Doerr, Carola Doerr, and Johannes Lengler. 2021. Self-Adjusting Mutation Rates with Provable Optimal Success Rules. Algorithmica 83, 10 (2021), 3108-3147. doi:10.1007/S00453-021-00854-3

[21] Benjamin Doerr and Carola Winzen. 2014. Playing Mastermind with constant-size memory. Theory of Computing Systems 55 (2014), 658-684.

[22] Carola Doerr and Markus Wagner. 2018. Simple on-the-fly parameter selection mechanisms for two classical discrete black-box optimization benchmark problems. In Proc. of Genetic and Evolutionary Computation Conference (GECCO), Hernán E. Aguirre and Keiki Takadama (Eds.). ACM, 943-950. doi:10.1145/3205455.3205560

[23] G Dulac-Arnold, R Evans, H van Hasselt, P Sunehag, T Lillicrap, J Hunt, T Mann, T Weber, T Degris, and B Coppin. [n. d.]. Deep reinforcement learning in large discrete action spaces. arXiv 2015. arXiv preprint arXiv:1512.07679 ([n. d.]).

[24] Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. 2019. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 (2019).

[25] Agoston Endre Eiben, Robert Hinterding, and Zbigniew Michalewicz. 1999. Parameter control in evolutionary algorithms. IEEE Transactions on Evolutionary Computation 3 (1999), 124-141.

[26] Brian W. Goldman and William F. Punch. 2015. Fast and Efficient Black Box Optimization Using the Parameter-less Population Pyramid. Evolutionary Computation 23 (2015), 451-479.

[27] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In ICML (Proceedings of Machine Learning Research, Vol. 80), Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 1856-1865. http://dblp.uni-trier.de/db/conf/icml/icml2018.html#HaarnojaZAL18

[28] Ji He, Mari Ostendorf, Xiaodong He, Jianshu Chen, Jianfeng Gao, Lihong Li, and Li Deng. 2016. Deep Reinforcement Learning with a Combinatorial Action Space for Predicting Popular Reddit Threads. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Jian Su, Kevin Duh, and Xavier Carreras (Eds.). Association for Computational Linguistics, Austin, Texas, 1838-1848. doi:10.18653/v1/D16-1189

[29] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.

[30] Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. 2022. On the role of discount factor in offline reinforcement learning. In International conference on machine learning. PMLR, 9072-9098.

[31] Riashat Islam, Peter Henderson, Maziar Gromokchi, and Doina Precup. 2017. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133 (2017).

[32] Giorgos Karafotias, Selmar K. Smit, and A. E. Eiben. 2012. A Generic Approach to Parameter Control. In Proc. of Applications of Evolutionary Computation (EvoApplications'12) (LNCS, Vol. 7248). Springer, 366-375. doi:10.1007/978-3-642-29178-4_37

[33] Stefan Kern, Sibylle D. Müller, Nikolaus Hansen, Dirk Büche, Jiri Ocenasek, and Petros Koumoutsakos. 2004. Learning probability distributions in continuous evolutionary algorithms - a comparative review. Natural Computing 3 (2004), 77-112.

[34] Stefan Kern, Sibylle D Müller, Nikolaus Hansen, Dirk Büche, Jiri Ocenasek, and Petros Koumoutsakos. 2004. Learning probability distributions in continuous evolutionary algorithms—a comparative review. Natural Computing 3 (2004), 77-112.

[35] Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations, (ICLR'15). Yoshua Bengio and Yann LeCun (Eds.).

[36] Per Kristian Lehre and Carsten Witt. 2012. Black-Box Search by Unbiased Variation. Algorithmica 64, 4 (2012), 623-642. doi:10.1007/S00453-012-9616-8

[37] Johannes Lengler and Simone Riedi. 2022. Runtime Analysis of the $(\mu+1)$-EA on the Dynamic BinVal Function. SN Comput. Sci. 3, 4 (2022), 324. doi:10.1007/S42979-022-01203-Z

[38] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Hees, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1509.02971

[39] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Mauro Birattari, and Thomas Stützle. 2016. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives 3 (2016), 43-58.

[40] Zeyuan Ma, Jiacheng Chen, Hongshu Guo, Yining Ma, and Yue-Jiao Gong. 2024. Auto-configuring exploration-exploitation tradeoff in evolutionary computation via deep reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference. 1497-1505.

[41] Volodymyr Mnih. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).

[42] Tai Nguyen, Phong Le, André Biedenkapp, Carola Doerr, and Nguyen Dang. 2025. On the Importance of Reward Design in Reinforcement Learning-based Dynamic Algorithm Configuration: A Case Study on OneMax with $(1 + (\lambda, \lambda))$-GA. In Proceedings of the Genetic and Evolutionary Computation Conference (NH Malaga Hotel, Malaga, Spain) (GECCO '25). Association for Computing Machinery, New York, NY, USA, 1162-1171. doi:10.1145/3712256.3726395

[43] Tai Nguyen, Phong Le, Carola Doerr, and Nguyen Dang. 2025. https://github.com/taindp98/OneMax-MPDAC.git.

[44] Faizan Rasheed, Kok-Lim Alvin Yau, Rafidah Md Noor, Celimuge Wu, and Yeh-Ching Low. 2020. Deep reinforcement learning for traffic signal control: A review. IEEE Access 8 (2020), 208016-208044.

[45] Ingo Rechenberg. 1973. Evolutionsstrategie. Friedrich Fromman Verlag (Günther Holzboog KG), Stuttgart.

[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).

[47] Michael A. Schumer and Kenneth Steiglitz. 1968. Adaptive step size random search. IEEE Trans. Automat. Control 13 (1968), 270-276.

[48] Mudita Sharma, Alexandros Komninos, Manuel López-Ibáñez, and Dimitar Kaza-kov. 2019. Deep reinforcement learning based parameter control in differential evolution. In Proceedings of the genetic and evolutionary computation conference. 709-717.

[49] David Speck, André Biedenkapp, Frank Hutter, Robert Mattmüller, and Marius Lindauer. 2021. Learning heuristic selection with dynamic algorithm configuration. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 31. 597-605.

[50] R.S. Sutton and A.G. Barto. 1998. Reinforcement Learning: An Introduction. IEEE Transactions on Neural Networks 9, 5 (1998), 1054-1054. doi:10.1109/TNN.1998.712192

[51] Arash Tavakoli, Fabio Pardo, and Petar Kormushev. 2018. Action branching architectures for deep reinforcement learning. In Proceedings of the aaaai conference on artificial intelligence, Vol. 32.

[52] Michele Tessari and Giovanni Iacca. 2022. Reinforcement learning based adaptive metaheuristics. In Proceedings of the Genetic and Evolutionary Computation Conference Companion. 1854-1861.
[53] Hado van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. In *Proceedings of the AAAI conference on artificial intelligence*, Vol. 30.

[54] Diederick Vermetten, Johannes Lengler, Dimitri Rusin, Thomas Bäck, and Carola Doerr. 2024. Empirical Analysis of the Dynamic Binary Value Problem with IOH-profiler. In *Proc. of Parallel Problem Solving from Nature (PPSN) (LNCS, Vol. 15149)*. Springer, 20–35. doi:10.1007/978-3-031-70068-2_2

[55] Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. *Machine learning* 8 (1992), 279–292.

[56] Ke Xue, Jiacheng Xu, Lei Yuan, Miqing Li, Chao Qian, Zongzhang Zhang, and Yang Yu. 2022. Multi-agent dynamic algorithm configuration. *Advances in Neural*

*Information Processing Systems* 35 (2022), 20147–20161.

[57] Wenjie Yi and Rong Qu. 2023. Automated design of search algorithms based on reinforcement learning. *Information Sciences* 649 (2023), 119639.

[58] Wenjie Yi, Rong Qu, Licheng Jiao, and Ben Niu. 2022. Automated design of metaheuristics using reinforcement learning within a novel general search framework. *IEEE Transactions on Evolutionary Computation* 27, 4 (2022), 1072–1084.

[59] Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Man-nor. 2018. Learn what not to learn: Action elimination with deep reinforcement learning. *Advances in neural information processing systems* 31 (2018).