# Benchmarking Powell's Legacy: Performance of Five Derivative-Free Solvers in pdfo on the bbob Test Suite

Dimo Brockhoff, Tanguy Villain

► To cite this version:

Dimo Brockhoff, Tanguy Villain. Benchmarking Powell's Legacy: Performance of Five Derivative-Free Solvers in pdfo on the bbob Test Suite. GECCO 2025 Companion: Genetic and Evolutionary Computation Conference Companion, Jul 2025, Malaga, Spain. pp.1833-1841, 10.1145/3712255.3734343. hal-05462326

HAL Id: hal-05462326

https://inria.hal.science/hal-05462326v1

Submitted on 16 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

Latest updates: https://dl.acm.org/doi/10.1145/3712255.3734343

RESEARCH-ARTICLE

# Benchmarking Powell's Legacy: Performance of Five Derivative-Free Solvers in pdfo on the bbob Test Suite

DIMO BROCKHOFF, INRIA Saclay, Palaiseau, France

TANGUY VILLAIN, CentraleSupélec - Paris-Saclay, Gif-sur-Yvette, Ile-de-France, France

Open Access Support provided by:

CentraleSupélec - Paris-Saclay

INRIA Saclay

<figure><img src="image_3.png" /></figure>

Published: 14 July 2025

Citation in BibTeX format

GECCO '25 Companion: Genetic and Evolutionary Computation Conference Companion

July 14 - 18, 2025

Malaga, Spain

Conference Sponsors:

SIGEVO
# Benchmarking Powell's Legacy: Performance of Five Derivative-Free Solvers in pdfo on the bbob Test Suite

Dimo Brockhoff

Tanguy Villain

Inria and CMAP, CNRS, Ecole Polytechnique, IP Paris
Palaiseau, France

CentraleSupélec and ESSEC
Paris campus, France

firstname.lastname@inria.fr

firstname.lastname@student-cs.fr

## ABSTRACT

The pdfo library by Tom M. Ragonneau and Zaikun Zhang makes the five derivative-free solvers BOBYQA, COBYLA, LINCOA, NEWUOA, and UOBYQA—originally written by Michael J. D. Powell— available in Python. In this paper, we are comparing their performance on the bbob test suite with three other solvers from the COCO data archive: CMA-ES from pycma, SLSQP and BFGS from scipy. We also compare the original solvers, written by Powell in Fortran 77, with the current pdfo versions, which saw multiple bug fixes and code improvements by Ragonneau and Zhang.

For the latter comparison, we do not see large effects on performance between the Fortran 77 version and the current pdfo version. The only notable exception is the Bent Cigar function where we observe differences by a factor of 2–5 for BOBYQA, LINCOA, and NEWUOA. Compared to the other baseline algorithms, BOBYQA, LINCOA and NEWUOA perform very similarly over all bbob functions, being about a factor of 5 slower than SLSQP and BFGS while UOBYQA—as the best-performing pdfo solver—outperforms SLSQP and BFGS for larger budgets when compared over all 24 bbob functions. The linear surrogate of COBYLA, on the contrary, is clearly worse over all functions than the other algorithms.

## CCS CONCEPTS

* Computing methodologies → Continuous space search.

## KEYWORDS

Benchmarking, Black-box optimization

### ACM Reference Format:

Dimo Brockhoff and Tanguy Villain. 2025. Benchmarking Powell's Legacy: Performance of Five Derivative-Free Solvers in pdfo on the bbob Test Suite. In *Genetic and Evolutionary Computation Conference (GECCO '25 Companion)*, July 14–18, 2025, Malaga, Spain. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3712255.3734343

## 1 INTRODUCTION

Michael J. D. Powell has provided valuable Fortran implementations of his algorithms BOBYQA, COBYLA, LINCOA, NEWUOA, and UOBYQA that Tom M. Ragonneau and Zaikun Zhang make available in Python via their pdfo library [12]. In addition to the availability in Python (and Matlab) instead of Fortran only, Ragonneau and Zhang spent considerable time to find bugs in the original code

<figure><img src="image_4.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License. GECCO '25 Companion, July 14–18, 2025, Malaga, Spain

© 2025 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-1464-1/2025/07.

https://doi.org/10.1145/3712255.3734343

and to improve the currently available solvers in terms of robustness and efficiency [12]. Callable like any scipy optimize algorithm [15], they can be considered as practically relevant solvers when no derivative information is available (or useful).

With the exception of three NEWUOA variants [13], based on Powell's original Fortran implementation [9], none of Powell's other algorithms (nor any in the pdfo module) has been benchmarked so far on the bbob test suite [5] of the COCO platform [4]. The discussion of the pdfo solvers' performance on bbob is therefore the main purpose of this paper. We will not only compare the five pdfo solvers among each other but also put their performance into the larger perspective of the COCO data archive<fnref n="1" /> by comparing them to the archived results from solvers CMA-ES [2], BFGS [14], and SLSQP [14]. We will equally compare the original implementations of Powell (from the pdfo module) with the corrected and improved (default) versions of Ragonneau and Zhang.

## 2 THE PDFO ALGORITHMS OF POWELL

COBYLA is the first of the five pdfo algorithms, proposed by Powell in 1994 [7]. A trust-region algorithm, originally designed for solving “Constrained Optimization BY Linear Approximations”, it can be equally applied to unconstrained problems. In this case, the usually second step (after finding a feasible solution in the trust region in the algorithm’s first step) corresponds to optimizing a linear model of the objective function within the trust region. It is important to note that in COBYLA, the trust region radius is only decreased and never increased.

UOBYQA, published by Powell in 2002 [8], stands for “Unconstrained Optimization BY Quadratic Approximation” and fits perfectly into the setup of the derivative-free (or blackbox) optimization benchmarking framework of the COCO platform. UOBYQA can be seen as a typical trust-region algorithm (where in particular the trust region radius can be decreased and increased), with the restriction that a lower bound on the radius is ensured. The trust region radius’ lower bound by itself can only decrease with time based on certain conditions (see [12] or [8] for details).

NEWUOA, LINCOA, and BOBYQA are the latest of Powell's algorithms—tailored towards unconstrained, linearly constrained and bound-constrained derivative-free problems in particular [9, 10]<fnref n="2" />. All three algorithms are trust-region-based model-building algorithms but differ slightly in the way, the quadratic model's optimum is exploited. In particular, the algorithms exploit (or don't in the case of NEWUOA) the availability of linear or bound constraints.

<footnote n="1">¹https://coco-platform.org/data-archive/</footnote>

<footnote n="2">²Note that LINCOA has never been published by Powell in a scientific article.</footnote>
For more details on the internal workings of the algorithms, we refer the interested reader to the original papers by Powell or to [12] for a shorter overview.

# 3 EXPERIMENTAL PROCEDURE

## 3.1 Parameter Tuning

For our benchmarking experiments, we use the `pdfo` module from the Python package index `PyPI`, version 2.1.0, together with the `cocoex` and `cocopp` modules of `COCO`, version 2.8.1 and 2.7.3 respectively.

For all `pdfo` algorithms, we use the default setting except for the final trust region radius and the budget-based stopping criterion. For the trust region radius, we set the initial radius to 1 (as the default) and reduce the final radius to $10^{-15}$ in order to be able to solve the simplest of the `bbob` test functions to a precision of $10^{-8}$. The budget-based stopping criterion was firing only when the total number of $10^4 n$ function evaluations was exhausted. In the following, we distinguish between the algorithms as implemented in `Fortran 77` by Powell (with a suffix “-77”, called by setting the parameter ‘classical’ in `pdfo` to `True`) and the current version by Ragonneau and Zhang (suffix “-25”, the default in `pdfo`). Due to technical issues, most likely due to some of the known bugs with the original implementation containing infinite loops [12], we were not able to run `UOBYQA` in the `Fortran 77` version and removed it therefore from the comparison.

## 3.2 CPU Timing

In order to evaluate the CPU timing of the algorithm, we have run all five `pdfo` algorithms on 15 instances (actually on instances 1–15) of the entire `bbob` test suite [5] for $10^4 n$ function evaluations according to [6] and the current example experiment of `COCO` (including success-based restarts until 75% of all 15 instances are solved to a precision of $10^{-8}$, the total budget of $10^4 n$ function evaluations is exhausted or until maximally 20 restarts are performed). The Python code was run on a machine with 64 Intel Xeon E5-2683 v4 cpus with 2.10GHz on a single processor. The Python version was 3.10.8. The recorded times per function evaluation for all algorithms and dimensions can be found in Table 1.

# 4 RESULTS

Results from experiments according to [6] and [3] on the benchmark functions given in [5] are presented in Figures 1 (scaling of expected runtimes with dimension), 2 (aggregated results in subgroups, 5-D), 3 (aggregated results in subgroups, 20-D), and 4 (empirical runtime distributions per function, 20-D). Tables 2 and 3 show the expected runtimes (ERT) of all algorithms for certain targets and all functions (except for functions $f_3$ and $f_4$) in 20-D as complement of Figure 4. The reason for not showing the results for $f_3$ and $f_4$ is space restrictions and the fact that the only finite number for those functions is the one for `CMA-ES` on function $f_3$ for the easiest target precision 10 (the value omitted being 15(8)). The entire table, together with the data in other dimensions not shown here, are made available on the supplementary material webpage https://brockho.github.io/bbob2025-SupplementaryMaterial/.

<table><caption>Table 1: Median runtimes of the benchmarking experiment per function evaluation in milliseconds. Algorithms ending with 25 correspond to the current `pdfo` implementation, algorithms ending with 77 correspond to the original `Fortran 77` code of Powell. Note that the experiments with the original `UOBYQA-77` did not produce any output.</caption><thead><tr><th></th><th>2-D</th><th>3-D</th><th>5-D</th><th>10-D</th><th>20-D</th><th>40-D</th></tr></thead><tbody><tr><td>BOBYQA-25</td><td>0.17</td><td>0.11</td><td>0.067</td><td>0.040</td><td>0.054</td><td>0.14</td></tr><tr><td>COBYLA-25</td><td>0.12</td><td>0.068</td><td>0.040</td><td>0.027</td><td>0.032</td><td>0.092</td></tr><tr><td>LINCOA-25</td><td>0.14</td><td>0.10</td><td>0.067</td><td>0.040</td><td>0.049</td><td>0.11</td></tr><tr><td>NEWUOA-25</td><td>0.14</td><td>0.10</td><td>0.064</td><td>0.043</td><td>0.050</td><td>0.11</td></tr><tr><td>UOBYQA-25</td><td>0.17</td><td>0.16</td><td>0.18</td><td>0.42</td><td>1.5</td><td>7.7</td></tr><tr><td>BOBYQA-77</td><td>0.16</td><td>0.11</td><td>0.064</td><td>0.040</td><td>0.052</td><td>0.14</td></tr><tr><td>COBYLA-77</td><td>0.12</td><td>0.066</td><td>0.038</td><td>0.026</td><td>0.031</td><td>0.095</td></tr><tr><td>LINCOA-77</td><td>0.14</td><td>0.099</td><td>0.063</td><td>0.038</td><td>0.044</td><td>0.094</td></tr><tr><td>NEWUOA-77</td><td>0.15</td><td>0.10</td><td>0.065</td><td>0.042</td><td>0.048</td><td>0.11</td></tr></tbody></table>

The experiments themselves were performed with `COCO [4]`, version 2.8.1 of `cocoex`, the plots were produced with `cocopp`, version 2.7.3.

The **expected runtime (ERT)** depends on a given target precision, $I_{\text{target}} = f_{\text{opt}} + \Delta f$, and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach $f_t$, summed over all trials and divided by the number of trials that actually reached $f_t$ [4, 11]. **Statistical significance** is tested with the rank-sum test for a given target $\Delta f$ using, for each trial, either the number of needed function evaluations to reach $f_t$ (inverted and multiplied by -1), or, if the target was not reached, the best $\Delta f$-value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial under consideration.

In the following, we first compare the five `pdfo` algorithms with three baseline algorithms from the `COCO` data archive: `CMA-ES-2019` from [2] as well as `BFGS-scipy` and `SLSQP-scipy` from [14]. Second, we compare the current implementations of four `pdfo` algorithms with their `Fortran 77` version, implemented by Powell.

## 4.1 The Current Implementations

When looking at the overall results, aggregated over all `bbob` functions, we can assign the five `pdfo` algorithms into three “groups”. `COBYLA` shows the worst performance with its linear surrogate model clearly not able to solve as many function/target pairs as the other algorithms, although it is comparable to the other algorithms for small budgets (and even better for small budgets on the moderate and weakly-structured multimodal functions). `BOBYQA`, `LINCOA`, and `NEWUOA` behave quite similarly with a medium performance—clearly better than `COBYLA` for large budgets on all function groups in 5-D and, in 20-D, better than `COBYLA` only on the separable, moderate, and ill-conditioned functions (while the performance is similar for all four algorithms in 20-D). The fifth `pdfo` algorithm, `UOBYQA`, behaves entirely different: for small budgets, `UOBYQA` is slower than the other `pdfo` algorithms, indicating how much can be gained by the underdetermined quadratic interpolation, introduced by Powell in `BOBYQA`, `LINCOA` and `NEWUOA`. Overall, `UOBYQA` behaves closer to `CMA-ES` while for the largest
<figure><img src="image_5.png" /><figcaption>**Figure 1:** Expected running time (ERT in number of f-evaluations as $\log_{10}$ value), divided by dimension for target function value $10^{-8}$ versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of $f_1$ and $f_{24}$. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a better result compared to all other algorithms with $p < 0.01$ and Bonferroni correction number of dimensions (six). Legend: ◦: BFGS-scipy-2019 Varelas, ♦: CMA-ES-2019 Hansen, ⋆: SLSQP-scipy-2019 Varelas, +: bobyqa-25 of pdfo-maxfevIsBudget-FTRR1e-15 10000D on bbob, ▽: cobyla-25 of pdfo-maxfevIsBudget-FTRR1e-15 10000D on bbob, x: lincoa-25 of pdfo-maxfevIsBudget-FTRR1e-15 10000D on bbob, △: newuoa-25 of pdfo-maxfevIsBudget-FTRR1e-15 10000D on bbob, ◊: uobyqa-25 of pdfo-maxfevIsBudget-FTRR1e-15 10000D on bbob-0001.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Figure 2: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 5-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 3: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 20-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 4: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets 10[−8..2] in dimension 20.</figcaption></figure>

budgets being worse than CMA-ES but better than the other pdfo algorithms from about 500n function evaluations in 20-D and earlier in smaller dimension.

It is remarkable that, in the overall ECDFs, UOBYQA is never slower than a factor of 10 compared to any other algorithm in the comparison, except for CMA-ES for the harder targets (when UOBYQA is the second-best algorithm among the ones shown here) and in the very beginning. Similar or even better results can be observed for smaller dimensions: the smaller the dimension, the

earlier UOBYQA shows better performance than the other pdfo algorithms. In dimension 2 it is even the best or slightly slower than the best for all budgets (results shown in the supplementary material only).

Let us now have a look at some results on some single functions. With the exception of COBYLA, all pdfo algorithms solve the simplest, unimodal functions of bbob in all dimensions to the highest precision of 10⁻⁸: sphere, separable and rotated ellipsoid, attractive sector sphere, discus, and bent cigar as well as the (not unimodal)
<table><caption>Table 3: Continuation of Table 2, showing ERT ratios to the best2009 reference algorithm. Also here, if rows are removed due to limited space, all entries are infinite, the budget in the last column is 2e6 and the success probability is 0 of 15 instances. Data</caption><thead><tr><th colspan="3">Δf<sub>opt</sub></th><th colspan="3">Δf<sub>opt</sub></th><th colspan="3">Δf<sub>opt</sub></th><th colspan="3">Δf<sub>opt</sub></th></tr><tr><th></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>1e-8</th><th>1e-1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>1e-8</th></tr><tr><th></th><th>f21</th><th>561</th><th>6541</th><th>14103</th><th>14318</th><th>14643</th><th>15567</th><th>17589</th><th>f23</th><th>3.0</th><th>1614</th><th>67457</th><th>3.7e5</th><th>4.9e5</th><th>8.1e5</th><th>8.4e5</th><th>15/15</th></tr></thead><tbody><tr><td>ubobyqa</td><td>3.5(4)</td><td>2.6(3)</td><td>1.8(2)</td><td>1.8(2)</td><td>1.8(2)</td><td>1.7(2)</td><td>1.5(1)</td><td>1.5(1)</td><td>2.3(2)</td><td>1.9(2)</td><td>12(14)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>3509</td><td>3/15</td></tr><tr><td>newuoa</td><td>1.6(2)</td><td>1.2(1)</td><td>1.3(1)</td><td>1.3(1)</td><td>1.2(1)</td><td>1.2(1)</td><td>1.0(1)</td><td>1.0(1)</td><td>1.9(2)</td><td>3.3(4)</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2248</td><td>3/15</td></tr><tr><td>lincoa</td><td>1.3(2)</td><td>0.71(0.9)</td><td>0.61(0.7)</td><td>0.60(0.6)</td><td>0.59(0.5)</td><td>0.57(0.6)</td><td>0.51(0.6)</td><td>0.51(0.6)</td><td>2.1(2)</td><td>1.6(2)</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>1355</td><td>3/15</td></tr><tr><td>cobyla</td><td>2.1(3)</td><td>2.0(3)</td><td>1.8(2)</td><td>1.8(2)</td><td>1.8(2)</td><td>1.7(2)</td><td>1.5(2)</td><td>1.4(2)</td><td>2.4(3)</td><td>5.3(6)</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>850</td><td>3/15</td></tr><tr><td>bobyqa</td><td>1.2(2)</td><td>1.8(2)</td><td>1.6(2)</td><td>1.6(2)</td><td>1.6(2)</td><td>1.5(2)</td><td>1.5(2)</td><td>1.4(2)</td><td>2.1(2)</td><td>3.5(4)</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>894</td><td>3/15</td></tr><tr><td>CMA-ES</td><td>4.7(11)</td><td>121(207)</td><td>62(72)</td><td>62(115)</td><td>61(56)</td><td>57(90)</td><td>51(65)</td><td>51(65)</td><td>1.9(2)</td><td>1064(1978)</td><td>154(176)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>76e</td><td>0/15</td></tr><tr><td>BFGS-sc</td><td>1.5(2)</td><td>1.5(2)</td><td>1.1(1)</td><td>1.1(1)</td><td>1.1(1)</td><td>1.0(1)</td><td>15(16)</td><td>15(16)</td><td>1.9(2)</td><td>4117(3457)</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e6</td><td>0/15</td></tr><tr><td>SLSQP-s</td><td>0.68(1)</td><td>0.95(0.9)</td><td>1.0(1)</td><td>0.99(1)</td><td>0.97(1)</td><td>0.93(1)</td><td>0.92(1)</td><td>1.3e5</td><td>1.9(2)</td><td>189(416)</td><td>16(10)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e6</td><td>0/15</td></tr><tr><td>f22</td><td>467</td><td>5580</td><td>23491</td><td>24163</td><td>24948</td><td>26847</td><td>24948</td><td>24948</td><td>1.3e6</td><td>7.5e6</td><td></td><td>5.2e7</td><td>5.2e7</td><td>5.2e7</td><td>5.2e7</td><td>5.2e7</td><td>3/15</td></tr><tr><td>ubobyqa</td><td>4.9(7)</td><td>3.3(4)</td><td>4.0(4)</td><td>3.9(4)</td><td>3.8(5)</td><td>3.6(4)</td><td>0.73(0.8)</td><td>0.73(0.8)</td><td>7/300</td><td>∞</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2538</td><td>3/15</td></tr><tr><td>newuoa</td><td>2.6(3)</td><td>1.8(2)</td><td>4.8(5)</td><td>4.6(5)</td><td>4.5(5)</td><td>4.2(4)</td><td>0.85(1.0)</td><td>0.85(1.0)</td><td>3/300</td><td>∞</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>1584</td><td>3/15</td></tr><tr><td>lincoa</td><td>2.2(3)</td><td>1.4(1)</td><td>2.5(3)</td><td>2.5(3)</td><td>2.4(2)</td><td>2.3(2)</td><td>0.45(0.5)</td><td>0.45(0.5)</td><td>4/300</td><td>∞</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>1566</td><td>3/15</td></tr><tr><td>cobyla</td><td>4.4(7)</td><td>3.9(4)</td><td>5.8(6)</td><td>5.7(6)</td><td>5.6(6)</td><td>5.3(6)</td><td>1.1(1)</td><td>0.53(0.6)</td><td>4/300</td><td>∞</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>5304</td><td>3/15</td></tr><tr><td>bobyqa</td><td>2.1(3)</td><td>1.7(2)</td><td>3.0(4)</td><td>2.9(4)</td><td>2.8(4)</td><td>2.6(3)</td><td>∞</td><td>1e6</td><td>5/300</td><td>∞</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>1544</td><td>3/15</td></tr><tr><td>CMA-ES</td><td>6.6(13)</td><td>156(163)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>1e6</td><td>0/15</td><td>16(20)</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>5e6</td><td>0/15</td></tr><tr><td>BFGS-sc</td><td>5.7(11)</td><td>3.9(6)</td><td>12(25)</td><td>12(25)</td><td>12(24)</td><td>11(22)</td><td>22(35)</td><td>22(35)</td><td>0/15</td><td>10(10)</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e6</td><td>0/15</td></tr><tr><td>SLSQP-s</td><td>2.9(6)</td><td>3.5(6)</td><td>8.7(11)</td><td>8.5(11)</td><td>8.2(11)</td><td>7.7(10)</td><td>6.3(10)</td><td>6.3(10)</td><td>1/15</td><td>∞</td><td></td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e6</td><td>0/15</td></tr></tbody></table>

produced with COCO v2.7.3

linear slope function. Also the separable and rotated Rosenbrock functions are solved, as are the Gallagher multi-peaks (except $f_{22}$ in 40-D). Interestingly, the sum of different powers function is only solved by UOBYQA (and CMA-ES), although with a slower and slower convergence for the hardest targets in dimensions larger than 5. With a larger budget, it is, though, expected that by restarting the algorithms often enough, also the other pdfo algorithms will solve the different powers function.

For the multi-modal functions—with the exception of the Gallagher functions—all five pdfo solvers show difficulties to solve more than 30% of the targets in 20-D (slightly higher percentages of the targets are solved in lower dimension by some algorithms). It is important to note that, if one of the pdfo algorithms outperforms the others on a multi-modal function, it is always UOBYQA (not for all dimensions and for all functions though). This is most pronounced on the Weierstrass (except in 2-D), both Schaffer (except in 2-D and 20-D) and Katsuura functions.

Overall, we can conclude that especially UOBYQA performs well on many functions, often comparable to SLSQP, BFGS, and CMA-ES but it also shows problems when solving multi-modal problems.

## 4.2 Comparison with Powell's Code

Figure 5 shows COCO's empirical runtime distributions for all pdfo algorithms (except UOBYQA<fnref n="3" />) in dimension 20 for all 24 single functions, all five function groups and aggregated over all functions. All results in other dimensions, scaling plots and tables can be found as well on the supplementary material webpage.

On a global scale, we see no big differences between the original Powell implementation and the improved versions within pdfo. NEWUOA, LINCOA, and BOBYQA behave very similarly when aggregated over all functions and COBYLA performing worse for all budgets. The differences between the original Fortran 77 versions and the current default version of the algorithms in pdfo are smaller than the differences between different algorithms. The only exception is NEWUOA for which we see a slight improvement from the original version to its current one in the 50n to

100n budget range in 40-D (and less pronounced in smaller dimensions). This difference can be attributed almost solely to the Bent Cigar function ($f_{12}$) where we see an improvement of NEWUOA-25 over NEWUOA-77 in the same budget range. The other remarkable difference between the two algorithm versions can be observed for LINCOA where LINCOA-77 shows better performance over LINCOA-25 on the weakly structured multi-modal function group which comes mainly from the Gallagher 21 peaks problem ($f_{22}$)—an effect that must be considered coming from (pseudo-)random properties of the objective function as this difference does not show up in other dimensions (and even shows the opposite effect in dimension 2). Contrary, the improved performance observed for NEWUOA-25 over NEWUOA-77 on the Bent Cigar function can be seen in other dimensions as well (not shown here due to space limitations).

## 5 CONCLUSIONS

The four algorithms BOBYQA, LINCOA, NEWUOA, and UOBYQA from the pdfo package show good to very good performance on the bbob test suite with UOBYQA performing best overall. The fifth pdfo algorithm, COBYLA, performs worse due to its weaker internal surrogate model (linear vs. quadratic), but unconstrained optimization is also not its focus: on the bbob-constrained test suite, COBYLA is a very fast algorithm, even more so when the number of constraints gets larger [1]. Our investigations on the differences between the original Fortran 77 implementation by Powell and the current default in pdfo show only minor differences. The only notable difference is the improved performance of the current BOBYQA, LINCOA, NEWUOA versions over the original one on the Bent Cigar where in dimensions 10 and 20, the performance gain is a factor of 2–5 over a wide range of budgets/targets while in lower dimension, the effect is smaller for NEWUOA and even reverted for BOBYQA and LINCOA. The efforts of Tom M. Ragonneau and Zaikun Zhang to improve, correct, and stabilize Powell's implementation, however, have the largest effect on UOBYQA which we did not manage to run on the bbob test suite in its original version while the current default of UOBYQA in pdfo shows the best performance among all five pdfo algorithms.

Finally, we would like to mention PRIMA [16], also developed by Zaikun Zhang and based on the work of pdfo, as an alternative to pdfo but leave the comparison with COCO for the future.

<footnote n="3">³Most likely due to known issues with the original code, the results for UOBYQA-77 could not be produced, although the call to pdfo was the same for all algorithms.</footnote>
<figure><img src="image_9.png" /><figcaption>Figure 5: Empirical runtime distributions, like in Figure 4 and Figure 3, but for comparing the current default versions (“-25”) of pdfo solvers with the original Fortran 77 implementations of Powell.</figcaption></figure>

## ACKNOWLEDGMENTS

We thank Zaikun Zhang for pointing us to pdfo in the first place and for sharing some valuable insides about his software. We also like to acknowledge the relation to COST Action CA22137 “Randomised Optimisation Algorithms Research Network” (ROAR-NET),

supported by COST (European Cooperation in Science and Technology), in particular in the context of WP3 “Single- and Multiobjective Optimisation”.

## REFERENCES

[1] Paul Dufossé and Asma Atamna. 2022. Benchmarking several strategies to update the penalty parameters in AL-CMA-ES on the bbob-constrained testbed.
In Genetic and Evolutionary Computation Conference (GECCO) Companion. ACM, New York, NY, USA, 1691–1699.

[2] Nikolaus Hansen. 2019. A global surrogate assisted CMA-ES. In Genetic and Evolutionary Computation Conference (GECCO 2019). ACM, New York, NY, USA, 664–672.

[3] Nikolaus Hansen, Anne Auger, Dimo Brockhoff, and Tea Tušar. 2022. Any-time Performance Assessment in Blackbox Optimization Benchmarking. IEEE Transactions on Evolutionary Computation 26, 6 (2022), 1293–1305.

[4] N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff. 2021. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. Optimization Methods and Software 36 (2021), 114–144. Issue 1. DOI: http://dx.doi.org/10.1080/10556788.2020.1808977

[5] N. Hansen, S. Finck, R. Ros, and A. Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Technical Report RR-6829. INRIA. http://hal.inria.fr/inria-00362633/en/

[6] N. Hansen, T. Tušar, O. Mersmann, A. Auger, and D. Brockhoff. 2016. COCO: The Experimental Procedure. ArXiv e-prints arXiv:1603.08776 (2016).

[7] Michael JD Powell. 1994. A direct search optimization method that models the objective and constraint functions by linear interpolation. In Advances in Optimization and Numerical analysis. Springer, Berlin, Heidelberg, 51–67.

[8] Michael JD Powell. 2002. UOBYQA: unconstrained optimization by quadratic approximation. Mathematical Programming 92, 3 (2002), 555–582.

[9] Michael JD Powell. 2006. The NEWUOA software for unconstrained optimization without derivatives. In Large-scale nonlinear optimization. Springer, Berlin, Heidelberg, 255–297.

[10] Michael JD Powell. 2009. The BOBYQA algorithm for bound constrained optimization without derivatives. Technical Report DAMTP 2009/NA06. University of

Cambridge, Cambridge, UK.

[11] Kenneth Price. 1997. Differential evolution vs. the functions of the second ICEO. In Proceedings of the IEEE International Congress on Evolutionary Computation. IEEE, Piscataway, NJ, USA, 153–157. DOI: http://dx.doi.org/10.1109/ICEC.1997.592287

[12] T. M. Ragonneau and Z. Zhang. 2023. PDFO: a cross-platform package for Powell's derivative-free optimization solvers. (2023). DOI: http://dx.doi.org/10.48550/arXiv.2302.13246 arXiv:2302.13246

[13] Raymond Ros. 2010. Comparison of NEWUOA with different numbers of interpolation points on the BBOB noiseless testbed. In Genetic and Evolutionary Computation Conference (GECCO) Companion. ACM, New York, NY, USA, 1487–1494.

[14] Konstantinos Varelas and Marie-Ange Dahito. 2019. Benchmarking multivariate solvers of SciPy on the noiseless testbed. In GECCO (Companion). ACM, New York, NY, USA, 1946–1954.

[15] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17 (2020), 261–272. DOI: http://dx.doi.org/10.1038/s41592-019-0686-2

[16] Z. Zhang. 2023. PRIMA: Reference Implementation for Powell's Methods with Modernization and Amelioration. available at http://www.libprima.net, DOI: 10.5281/zenodo.8052654. (2023).