# On the Robustness of BFGS to Positive and Negative Noise Outliers on the BBOB Test Suite

Alexandre Chotard, Anne Auger

► To cite this version:

Alexandre Chotard, Anne Auger. On the Robustness of BFGS to Positive and Negative Noise Outliers on the BBOB Test Suite. GECCO 2025 Companion - Genetic and Evolutionary Computation Conference Companion, Jul 2025, Malaga, Spain. pp.1850-1858, 10.1145/3712255.3734333. hal-05479493

HAL Id: hal-05479493

https://inria.hal.science/hal-05479493v1

Submitted on 28 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# On the Robustness of BFGS to Positive and Negative Noise Outliers on the BBOB Test Suite

Alexandre Chotard

alexandre.chotard@univ-littoral.fr

OSMOSE, LISIC, Université du Littoral de la Côte d'Opale

Calais, France

Anne Auger

anne.auger@inria.fr

Inria, CMAP, CNRS, Ecole Polytechnique, Institut

Polytechnique de Paris

Paris, France

## ABSTRACT

We investigate the impact of outlier noise on the performance of the scipy{ optimize implementation of the quasi-Newton BFGS solver. Using the BBOB testbed corrupted with positive—making solution look worse than what they are—or negative—making solutions look better than what they are—outliers simulated with a Cauchy distribution with a probability p, we analyze how the performance is impacted. We show that the impact of positive or negative noise outliers is almost symmetric, that on simple problems BFGS has some robustness to noise, and that for ill-conditionned problems BFGS appears to fail when p or the dimension is too high.

## CCS CONCEPTS

* Computing methodologies → Continuous space search.

## KEYWORDS

Benchmarking, Black-box optimization

### ACM Reference Format:

Alexandre Chotard and Anne Auger. 2025. On the Robustness of BFGS to Positive and Negative Noise Outliers on the BBOB Test Suite. In *Genetic and Evolutionary Computation Conference (GECCO '25 Companion)*, July 14–18, 2025, Malaga, Spain. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3712255.3734333

## 1 INTRODUCTION

The BFGS (Broyden–Fletcher–Goldfarb–Shanno) algorithm is a widely used iterative optimization algorithm for solving unconstrained non-linear optimization problems [4]. It belongs to the family of quasi-Newton methods, which approximate the Hessian matrix of second derivatives using gradient information only. When the gradient is not available, it can be approximated using finite differences and used for derivative-free (or black-box) optimization. The BFGS optimization algorithm is implemented in many optimization library and toolboxes including the scipy { optimize Python optimization toolbox. We benchmark here scipy { optimize’s implementation of BFGS on the BBOB testbed with positive then negative outliers, and evaluate using the COCO platform the impact of outliers on the performance of BFGS for different frequencies of occurrence of outliers.

More precisely, we consider additive noise where each noiseless BBOB function value f (x) is perturbed by adding with probability p_{add} the absolute value of a random variable with Cauchy distribution. With additive noise, candidate solutions appear worse than what they are on the noiseless function. We also consider negative noise where each noiseless BBOB function value f (x) is perturbed by subtracting with probability p_{sub} the absolute value of a Cauchy distribution. With negative noise, candidate solutions appear better than what they are. The Cauchy distribution is heavy-tailed and thus model some outliers.

By evaluating the effects of adding or subtracting noise, we can measure whether one effect has more impact on the performances of the algorithm.

The paper is organized as follows. In Section 2 we present the experimental procedure. In Section 3 we present the results, and provide a discussion in Section 4.

## 2 EXPERIMENTAL PROCEDURE

Following [1] and [3], scipy . optimize’s implementation of BFGS is ran on the BBOB test suite from the COCO platform [2] using COCO’s cocoex module, version 2.7.5. Our experiments are ran for dimension n ∈ {2, 3, 5, 10, 20}, with a budget of n × 10^3 function evaluations and 15 different instances for each problem of the testbed. The BFGS algorithm is used in a derivative-free mode where the gradient is estimated via finite-differences. On a noisy problem problem ˜f, given an initial solution x_0 provided by COCO’s ExperimentRepeater class, BFGS is ran with the command scipy . optimize . minimize( ˜f, x_0, method=’BFGS’, tol =1e-15, jac=’2-point’). The parameter tol stops the run if the estimated gradient has a L_\infty -norm lesser than tol. By default, the algorithm also terminates if its number of iterations is over 200 × n. Every iteration, the gradient is approximated using finite differences using the so-called 2-point method. The version of scipy used in this experiment is version 1.15.2.

Each BBOB problem f is wrapped by the COCO cocoex . noiser. Noisifier with parameters p_{add} and p_{sub} to a function ˜f such that

$$ \tilde{f}(x) := \begin{cases} f(x) + |\xi| & \text{with probability } p_{\text{add}}, \\ f(x) - |\xi| & \text{with probability } p_{\text{sub}}, \\ f(x) & \text{else.} \end{cases} $$

Note that the events of adding positive or negative noise cannot co-occur, and p_{add} + p_{sub} ≤ 1. More specifically in our experiments, either p_{add} or p_{sub} is set to 0, so that we observe only positive or only negative perturbations. Furthermore, the noise is so-called frozen, that is any number of evaluations for any x ∈ X will always return the same value ˜f (x). The noise ξ follows a standard Cauchy

<footnote>Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. Request permissions from owner/author(s).</footnote>

<footnote>GECCO '25 Companion, July 14–18, 2025, Malaga, Spain</footnote>

<footnote>© 2025 Copyright held by the owner/author(s). Publication rights licensed to ACM.</footnote>

<footnote>ACM ISBN 979-8-4007-1464-1/2025/07</footnote>

<footnote>https://doi.org/10.1145/3712255.3734333</footnote>
distribution multiplied by $a := \pi/2$. The coefficient $a$ is chosen such that the probability that $|\xi| \ge k$ is approximately $1/k$ for $k \ge 10$. To investigate the effect of noise, BFGS is ran on every BBOB problem with $p_{add} + p_{sub} \in \{0, 0.02, 0.05, 0.1, 0.2, 0.3, 0.4\}$.

## 3 RESULTS

The plots were produced with the cocopp module, version 2.7.2. Each curve is labelled with the corresponding $p_{add}$ or $p_{sub}$ value followed by ‘add’ or ‘sub’, depending on whether positive noise $|\xi|$ was added or subtracted. The ECDFs of the runtime distribution of the BFGS algorithm for dimension 5 (resp. dimension 20) aggregated over 51 targets on different groups of the 24 functions of the BBOB test suite are displayed in Figure 1 (resp. Figure 2). The detailed ECDFs for every function of the BBOB suite are provided for dimension 5 and positive outliers in Figure 3, for negative outliers in Figure 4, and similarly in dimension 20 in Figure 5 and Figure 6. Due to a lack of space, other dimensions are not shown.

The results show a high similarity between the performances for positive and for negative outliers. This may be unexpected: positive outliers are evaluated worse than they are, and usually are less perturbative than negative outliers, which can make an optimization algorithm select actually bad points or be misled to a bad direction. On most functions, the performances of BFGS against negative outliers still appear significantly worse than against positive outliers, with perhaps an exception on the ellipsoid function $f_{10}$ in 5-D as one can see between Figure 3 and Figure 4. In 20-D, looking at Figure 3 and Figure 4, it is not obvious that there is a difference on $f_{10}$. Overall, according to Figure 1 and Figure 2, the impact of negative outliers appears slightly larger on moderate and ill-conditionned functions.

As one may expect, the higher the outlier probability $p_{add}$ or $p_{sub}$ is, the higher is the difficulty of a problem. The rare cases where this is not obvious have few successes (such as on function 7 the step-ellipsoid), and what we see may be mostly due to stochastics. Overall, BFGS appears robust to perturbations with low probabilities, although increasing $p_{add}$ or $p_{sub}$ steadily decreases its performances. Still, on simple functions such as the sphere, if given enough time BFGS can reach all targets even with 40% of points perturbed. However, on ill-conditionned problems such as the separable ellipsoid $f_2$ or the rosenbrock function $f_8$ or more complex functions such as the attractive sector $f_6$, having more than 10% of perturbed points does appear to have a very large effect. This suggests that outliers, whether positive or negative, are particularly harmful in the estimation of the Hessian. Also, even on the simple sphere function, very low probabilities of perturbation still do significantly impede the performance of BFGS for reaching difficult targets.

When dimension increases, on simple functions the difficulty for noisy problems appear to scale similarly with dimension as without noise, while on functions for which high noise was a large problem such as $f_2$ and $f_6$, the difficulty increases much more with the dimension, and low outlier probabilities that had low impact in 5-D can have a much higher impact in 20-D.

## 4 DISCUSSION

Noisy problems have the reputation to be particularly difficult for gradient-based methods. Indeed, noise heavily perturbs the estimation of the gradient, which can in turn make line search fail (as line search assumes a descent direction) and stop the algorithm early. However, in our experiments, on simple enough functions such as the sphere function $f_2$ or the linear slope $f_6$, with enough restarts, BFGS still consistently solve the problems even with 0.4 perturbation probability. Still, even small noise probabilities do significantly slow the algorithm when reaching for difficult targets. When dimension increases, on these simple functions, for a given noise probability, the performances of BFGS appear to scale similarly as without noise. This may be unexpected: as dimension increases, the probability of the gradient estimate being perturbed by the noise increases as well, yet BFGS does not appear too impacted by perturbed gradients. Perturbations in the estimation of the gradient are massive, as when they occur the noise is multiplied by $\epsilon^{-1}$ where $\epsilon$ is the step used for estimating the gradient by finite difference, with $\epsilon$ being equal to approximately $10^{-8}$ in this setting. Yet it appears that in spite of this perturbation, the subsequent line-search is still successful enough to bring reliable improvement.

This does not hold for ill-conditioned or more complex functions such as the separable ellipsoid $f_2$, the attractive sector $f_6$ or the Rosenbrock function $f_8$, where probabilities of noise below some threshold $\kappa_{f,n}$ have small to negligible effect on the performances, and as outlier probabilities increase over $\kappa_{f,n}$, BFGS's performances deteriorate very fast. As the dimension $n$ increases, the value of $\kappa_{f,n}$ decreases and BFGS on these functions appear more and more sensitive to noise. As BFGS is affine-invariant, giving the right matrix at initialization to BFGS should yield the same performances on $f_1$ and $f_2$. This suggests that the difficulty with respect to additive noise has less to do with the perturbations on the estimation of the gradient, which has moderate impact on $f_1$, than the perturbations on BFGS's estimate of the Hessian. If we extrapolate this to higher dimensions, this could imply that on functions where learning the Hessian is critical for efficiency, BFGS may become unreliable for any probability of noise when the dimension is large enough. This could be an over-interpretation of the data, and more data in higher dimensions with higher budget would be useful.

Negative outliers are considered in general to be more difficult, and although the BFGS algorithm do show overall poorer performances with negative outliers than positive ones, the results are still highly similar between the two settings. This can be expected as when estimating the gradient through finite difference, positive or negative outliers have a similar averse effect, thus the quality of the gradient estimate should not be qualitatively too different. However, if the gradient estimate is not so perturbed as to make the line-search fail, positive outliers can make good points be mistakenly discarded, resulting in smaller steps and progress, whereas negative outliers can make the line-search believe that a bad point is good, which can be more harmful than slow progress.

## REFERENCES

[1] Nikolaus Hansen, Anne Auger, Dimo Brockhoff, and Tea Tušar. 2022. Anytime performance assessment in blackbox optimization benchmarking. *IEEE Transactions on Evolutionary Computation* 26, 6 (2022), 1293–1305.
<figure><img src="image_2.png" /><figcaption>Figure 1: Bootstrapped empirical cumulative distribution of the number of f-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 5-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 2: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (*FEvals/DIM*) for 51 targets with target precision in $10^{-8}...2$ for all functions and subgroups in 20-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 3: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of f-evaluations, divided by dimension (FEvals/DIM) for the 51 targets 10[−8..2] in dimension 5.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Figure 4: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets 10[−8..2] in dimension 5.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Figure 5: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of f-evaluations, divided by dimension (FEvals/DIM) for the 51 targets $10^{[-8..2]}$ in dimension 20.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Figure 6: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of f-evaluations, divided by dimension (FEvals/DIM) for the 51 targets 10[−8..2] in dimension 20.</figcaption></figure>
[2] N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff. 2021. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. *Optimization Methods and Software* 36 (2021), 114–144. Issue 1. doi:10.1080/10556788.2020.1808977

[3] Nikolaus Hansen, Tea Tusar, Olaf Mersmann, Anne Auger, and Dimo Brockhoff. 2016. COCO: The experimental procedure. *arXiv preprint arXiv:1603.08776* (2016).

[4] Jorge Nocedal and Stephen J Wright. 1999. *Numerical optimization*. Springer.