# Benchmarking CMA-ES under Additive and Subtractive Noise on the BBOB Testbed

Oskar Girardin

► To cite this version:

Oskar Girardin. Benchmarking CMA-ES under Additive and Subtractive Noise on the BBOB Testbed. GECCO '25 Companion: Proceedings of the Genetic and Evolutionary Computation Conference Companion, Jul 2025, Malaga, Spain. pp.1867-1874, 10.1145/3712255.3734332. hal-05406464

HAL Id: hal-05406464

https://hal.science/hal-05406464v1

Submitted on 9 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Benchmarking CMA-ES under Additive and Subtractive Noise on the BBOB Testbed

Oskar Girardin

Inria, École polytechnique, Institut Polytechnique de Paris

Palaiseau, France

firstname.lastname@inria.fr

## Abstract

We benchmark a non-elitist CMA-ES algorithm on the BBOB testbed with additive and subtractive noise. In particular, we consider the case where re-evaluated solutions produce the same observed function value. As a comparison, we benchmark a version of CMA-ES with resampling, which aims at reducing the effective noise level. We find CMA-ES to be more sensitive to subtractive noise than to additive noise in dimensions 2, 3, 5, 10, 20 and 40. Resampling for CMA-ES appears to be detrimental for low noise levels, while it is beneficial for high noise levels.

## CCS Concepts

* Computing methodologies → Continuous space search.

## Keywords

Benchmarking, Black-box optimization, Randomized optimization

## ACM Reference Format:

Oskar Girardin. 2025. Benchmarking CMA-ES under Additive and Subtractive Noise on the BBOB Testbed. In *Genetic and Evolutionary Computation Conference (GECCO '25 Companion), July 14–18, 2025, Malaga, Spain*. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3712255.3734332

## 1 Introduction

The covariance matrix adaptation evolution strategy (CMA-ES) [3] is a stochastic, population-based search method that aims at minimizing a black-box function $f: \mathbb{R}^n \to \mathbb{R}$. As a heuristic for its search, it uses the ranking of solutions with respect to the objective function. Notably, no gradient information is used or required. The motivation of this paper is to test the robustness of CMA-ES to heavy-tailed "frozen" noise, which is a difficult setting for an algorithm to operate in. While a noisy bbob test suite exists [2], we consider a noise wrapper around the 24 bbob functions, which adds (or subtracts) heavy-tailed half-Cauchy noise instead of Gaussian noise.

## 2 Experimental Procedure

The test suite used in our experiments is a modified version of the full bbob test suite [7]. The test suite has the following addition. Whenever a solution $x \in \mathbb{R}^n$ is evaluated on a test function, a non-negative noise term $\epsilon \in \mathbb{R}^+$ is added to or subtracted from the true

function value with probabilities $p_{add}$ and $p_{sub}$, respectively. The resulting observed objective function $\tilde{f}$ is shown in Equation (1).

$$ \tilde{f}(x) := \begin{cases} f(x) + \epsilon, & \text{with probability } p_{add} \\ f(x) - \epsilon, & \text{with probability } p_{sub} \end{cases} \quad (1) $$

The noise term $\epsilon$ is sampled from a standard half-Cauchy distribution multiplied by $\pi/2$. It has the property that for $k \ge 10$ the probability of $\epsilon > k$ is approximately $1/k$, making it heavy-tailed. Note that additive and subtractive noise cannot be nonzero simultaneously, that is $p_{add} \times p_{sub} = 0$.

This setup has the particularity that when re-evaluating a solution, it will produce (exactly) the same noise term $\epsilon$ and function value $\tilde{f}$, making the noise effectively "frozen". Depending on the sign of the noise term, such an outlier will look worse (positive noise) or better (negative noise) than it actually is. This is a challenge for an algorithm navigating through the search space. The used implementation of this noise model is the Noisifier class in the coco-experiment module.

We benchmark the $(\mu/\mu_W, \lambda)$-aCMA-ES algorithm [3], where $\mu$ corresponds to the number of parents, which we combine via weighted recombination and $\lambda$ is the population size. We use the default population size $\lambda = 4+3 \log(n)$ and negative weights for the worst solutions ("active" adaptation). The algorithm is run without restarts and we set the initial step-size $\sigma_0 = 2$. We use the Python implementation GitHub/CMA-ES/pycma [4], version 4.0.0 and its default termination criteria.

In addition to the default setup, we benchmark CMA-ES with the following simple noise handling module. To determine the ranking of a solution $x \in \mathbb{R}^n$, we additionally sample 2 independent vectors $v_1, v_2 \in \mathbb{R}^n$ with distribution $N(0, (10^{-5} \times \sigma)^2C)$. The associated function value of $x$ used for selection is the median of $f(x), f(x+v_1)$ and $f(x+v_2)$. While this triples the number of evaluations per iteration, this can be useful against noisy outliers, as this reduces the effective noise level observed by the algorithm (for $p_{add}, p_{sub} < 0.5$).

The experiments were performed with COCO [6], version 2.7.5, the plots were produced with version 2.7.3. The budget used throughout the experiments is $10^5 \times n$ function evaluations. We test problems in dimension 2, 3, 5, 10, 20 and 40, with 15 instances per function. We consider 51 targets with 10 target precisions ranging from $10^{-8}$ to $10^2$. While the budget is not exhausted and less than 75% of instances are solved, the algorithm is run again on all 15 function instances (maximally 20 restarts). Experiments were conducted according to [8] and [5] on the benchmark functions described in [1]. The tested noise levels are $p_{add}, p_{sub} \in \{0.0, 0.02, 0.05, 0.1, 0.2, 0.3, 0.4\}$.

<figure><img src="image_2.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License. GECCO '25 Companion, July 14–18, 2025, Malaga, Spain

© 2025 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-1464-1/2025/07

https://doi.org/10.1145/3712255.3734332
## 3 CPU Timing

The C/Python code was run on an AMD EPYC 7702 CPU @ 2GHz with 2 processors and 64 cores. Table 2 shows the total time of each experiment and Table 1 the time per function evaluation without noise handling. We notice that subtractive noise experiments take longer than additive noise experiments. This is explained by the fact that individual runs are longer on average for subtractive noise.

<table><caption>Table 1: Time per function evaluation (in seconds) for additive and subtractive noise by dimension for noise levels 0.0, 0.02, and 0.4 without noise handling.</caption><thead><tr><th rowspan="2">Dim.</th><th colspan="3">p<sub>add</sub> (with p<sub>sub</sub> = 0)</th><th colspan="3">p<sub>sub</sub> (with p<sub>add</sub> = 0)</th></tr><tr><th>0</th><th>0.02</th><th>0.4</th><th>0</th><th>0.02</th><th>0.4</th></tr></thead><tbody><tr><td>2</td><td>1.0e-4</td><td>1.2e-4</td><td>1.1e-4</td><td>1.0e-4</td><td>1.1e-4</td><td>1.1e-4</td></tr><tr><td>3</td><td>8.8e-5</td><td>1.0e-4</td><td>1.0e-4</td><td>8.8e-5</td><td>9.8e-5</td><td>1.0e-4</td></tr><tr><td>5</td><td>8.2e-5</td><td>9.6e-5</td><td>9.7e-5</td><td>8.3e-5</td><td>9.3e-5</td><td>9.9e-5</td></tr><tr><td>10</td><td>7.4e-5</td><td>8.7e-5</td><td>8.9e-5</td><td>7.4e-5</td><td>8.5e-5</td><td>9.2e-5</td></tr><tr><td>20</td><td>7.3e-5</td><td>8.6e-5</td><td>8.8e-5</td><td>7.3e-5</td><td>8.4e-5</td><td>9.2e-5</td></tr><tr><td>40</td><td>8.3e-5</td><td>9.5e-5</td><td>9.7e-5</td><td>8.2e-5</td><td>9.2e-5</td><td>1.0e-4</td></tr></tbody></table>

<table><caption>Table 2: Total time of experiments by noise level (aggregated across dimensions) with a budget of $10^5 \times n$ function evaluations for both algorithms.</caption><thead><tr><th rowspan="2">Noise level</th><th colspan="2">Without noise handling</th><th colspan="2">With noise handling</th></tr><tr><th>Additive</th><th>Subtractive</th><th>Additive</th><th>Subtractive</th></tr></thead><tbody><tr><td>0.0</td><td>2h58</td><td>3h00</td><td>4h37</td><td>4h08</td></tr><tr><td>0.02</td><td>3h29</td><td>3h54</td><td>5h18</td><td>5h45</td></tr><tr><td>0.05</td><td>3h36</td><td>4h11</td><td>5h03</td><td>5h32</td></tr><tr><td>0.1</td><td>3h40</td><td>4h47</td><td>5h27</td><td>5h55</td></tr><tr><td>0.2</td><td>3h55</td><td>7h24</td><td>5h46</td><td>6h41</td></tr><tr><td>0.3</td><td>4h17</td><td>14h22</td><td>5h57</td><td>16h03</td></tr><tr><td>0.4</td><td>4h40</td><td>10h03</td><td>6h49</td><td>20h28</td></tr></tbody></table>

## 4 Results

In Figure 1a and Figure 1b, we see the empirical cumulative distribution functions (ECDFs) by function groups in dimension 20 for additive and subtractive noise respectively for CMA-ES without noise handling. Figure 2a and Figure 2b show the same quantities for CMA-ES with the noise handling module. The pairs of figures 3, 4 and 5, 6 show function-wise ECDFs in dimension 20 for both noise types for CMA-ES respectively with and without noise handling.

### 4.1 Default CMA-ES

**Additive noise.** In our experiments, additive noise has a moderate effect on the performance of CMA-ES on the bbob test suite. In the case of functions that CMA-ES is already able to solve confidently, a high noise level leads to a comparatively small slowdown. Indeed, on $f_1$ (sphere), CMA-ES requires approximately twice as many function evaluations with $p_{add} = 0.4$ compared to the noiseless case. Other examples of functions where CMA-ES is only slightly affected are $f_2$, $f_5$, $f_{10}$ and $f_{11}$. Functions that are difficult to solve for CMA-ES in the noiseless case, remain difficult to solve with noise,

which is expected. It is the case for $f_3$, $f_4$, $f_7$, $f_{15}$, $f_{16}$, $f_{19}$, $f_{20}$, $f_{23}$ and $f_{24}$ in dimension 20 and $f_4$ and $f_{24}$ in dimension 5 (not shown). In dimension 20 for a noise level of 0.4, only a single function ($f_{13}$) becomes unsolved due to noise.

**Subtractive noise.** Subtractive noise has a considerably larger effect on the performance of CMA-ES. In dimension 5 (not shown), CMA-ES solves only 2 out of 24 test functions with a noise level of 0.4, compared to 18 of 24 functions for additive noise. In dimension 20 for a noise level of 0.3 and 0.4, 12 out of 24 functions are unsolved due to noise.

### 4.2 CMA-ES with resampling

**Additive noise.** Additive noise has little impact on the performance of CMA-ES with noise handling. As expected, resampling represents a significant computational cost. As an example, the 20D-Sphere in the noiseless case is solved in under 500 evaluations with resampling, whereas it takes at most 150 evaluations without resampling. For a noise level of 0.4, this function is solved in under 650 and 250 evaluations respectively. This corresponds to an increase in evaluations of roughly 66% without resampling and 30% with resampling due to noise.

Generally, additive noise has a small effect on the performance of CMA-ES with noise handling. The only case where additive noise causes failure in 20D is $f_{13}$, with a noise level of 0.4.

**Subtractive noise.** Adding noise handling improves performance for noise level 0.3 and a decreases it for noise levels up to 0.1. For a noise level of 0.3, the default CMA-ES only solves a single 20D-function, while the CMA-ES with resampling solves 10 out of 24 problems. In general, there is a tradeoff between reducing effective noise and the cost of re-evaluations. When there is much noise, the former dominates, while the latter dominates for little noise. In our experiments, we find the threshold between the two effects to lie between 0.1 and 0.3. We expect this quantity to increase by increasing the number of re-evaluations.

## 5 Conclusion

We benchmarked the pycma implementation [4] of $(\mu/\mu_W, \lambda)$-aCMA-ES without restarts on a noise wrapper around bbob with and without resampling solutions. We tested additive and subtractive noise in dimensions 2, 3, 5, 10, 20 and 40 with a budget of $10^5 \times n$ using COCO [6]. We found CMA-ES to be more resilient to bad outliers than to good outliers. This is expected, as bad outliers do not play a significant role in the selection process of (CMA-)ES. In contrast, good outliers are attributed large weights for recombination and the algorithm is perturbed by random directions. This slows down the algorithm. Resampling solutions and taking the median function value mitigates the effect of noise by reducing the effective noise level. This increases the number of function evaluations per iteration.

## References

[1] S. Finck, N. Hansen, R. Ros, and A. Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Presentation of the Noiseless Functions. Technical Report 2009/20. Research Center PPE. http://coco.lri.fr/downloads/download15.03/bbobdocfunctions.pdf Updated February 2010.

[2] S. Finck, N. Hansen, R. Ros, and A. Auger. 2010. Real-Parameter Black-Box Optimization Benchmarking 2010: Presentation of the Noisy Functions. Technical Report 2010/10. Technical University of Denmark. http://www.dtu-d2c.com/ptf Downloaded from http://www.dtu-d2c.com/ptf by 192.168.0.1 on 2010-12-10 15:11:00.
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /><figcaption>(a) CMA-ES without noise handling on bbob with added noise (bad outliers): Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{-8..2}$ for all functions and subgroups in 20-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /><figcaption>(b) CMA-ES without noise handling on bbob with subtracted noise (good outliers): Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{-8..2}$ for all functions and subgroups in 20-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>
<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /><figcaption>(a) CMA-ES with noise handling on bbob with added noise (bad outliers): Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{-8.2}$ for all functions and subgroups in 20-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /><figcaption>(b) CMA-ES with noise handling on bbob with subtracted noise (good outliers): Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{-8.2}$ for all functions and subgroups in 20-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>
<figure><img src="image_27.png" /><figcaption>Figure 3: CMA-ES without noise handling on bbob with added noise (bad outliers): Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets $10^{-8 \dots 2}$ in dimension 20.</figcaption></figure>

Report 2009/21. Research Center PPE. http://numbbo.github.io/gforge/bbob2010-downloads

[3] N. Hansen. 2016. The CMA Evolution Strategy: A Tutorial. CoRR abs/1604.00772 (2016), 39 pages. arXiv:1604.00772 http://arxiv.org/abs/1604.00772

[4] N. Hansen, Y. Akimoto, and P. Baudis. 2019. CMA-ES/pycma on Github. Zenodo. DOI:10.5281/zenodo.2559634. doi:10.5281/zenodo.2559634

[5] N. Hansen, A. Auger, D. Brockhoff, and T. Tušar. 2022. Anytime Performance Assessment in Blackbox Optimization Benchmarking. *IEEE Transactions on Evolutionary Computation* 26, 6 (2022), 1293–1305.

[6] N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff. 2021. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. *Optimization Methods and Software* 36 (2021), 114–144. Issue 1. doi:10.1080/10556788.2020.1808977

[7] N. Hansen, S. Finck, R. Ros, and A. Auger. 2009. *Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions*. Technical Report RR-6829. INRIA. http://hal.inria.fr/inria-00362633/en/

[8] N. Hansen, T. Tušar, O. Mersmann, A. Auger, and D. Brockhoff. 2016. COCO: The Experimental Procedure. *ArXiv e-prints* arXiv:1603.08776 (2016).
<figure><img src="image_28.png" /><figcaption>Figure 4: CMA-ES without noise handling on bbob with subtracted noise (good outliers): Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets $10^{-8 \dots 2}$ in dimension 20.</figcaption></figure>
<figure><img src="image_29.png" /><figcaption>Figure 5: CMA-ES with noise handling on bbob with additive noise (bad outliers): Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets 10[−8..2] in dimension 20.</figcaption></figure>
<figure><img src="image_30.png" /><figcaption>Figure 6: CMA-ES with noise handling on bbob with subtracted noise (good outliers): Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets $10^{-8 \dots 2}$ in dimension 20.</figcaption></figure>