CPU comparison of Nelder-Mead, Powell, and CMA-ES on non-convex functions
Domain: ml (optimization, numerical-methods)
Topic: Comparing gradient-free optimization algorithms (Nelder-Mead, Powell, CMA-ES) for non-convex benchmark functions using only CPU computation
Venue: ARC-Bench 2026
Headline metric: primary_metric (minimize)

Research briefing:
Gradient-free optimization methods are widely used when derivatives are
unavailable, unreliable, or expensive to compute. In low-to-medium
dimensional non-convex landscapes, direct-search methods such as
Nelder-Mead and Powell are frequently used because they are easy to call
through scipy.optimize. Population-based approaches such as CMA-ES can be
more robust to local minima and ill-conditioning, but they introduce extra
hyperparameters and potentially higher per-iteration cost.

A CPU-bounded benchmark can clarify practical trade-offs by evaluating these
methods under a fixed function-evaluation budget on standard synthetic
objective functions with known global minima. Rather than asking which method
is universally best, the relevant question is whether one method reaches
better objective values more reliably within the same budget and how much
runtime overhead that reliability costs.

A credible experiment should evaluate at least three optimization conditions
(Nelder-Mead, Powell, CMA-ES) across multiple non-convex test functions,
using repeated random starts and common stopping/budget rules. Reporting only
final objective value is incomplete; runtime and success-rate-to-threshold
should be included to capture both quality and efficiency. Statistical
summaries over seeds are required because single runs are high variance.

The resulting evidence should produce explicit pass/fail verdicts for each
hypothesis: whether CMA-ES improves best-found objective value, whether
Powell offers faster wall-clock convergence than CMA-ES at similar budgets,
and whether Nelder-Mead underperforms on multimodal landscapes. This keeps
the study measurable and feasible within a short single-core runtime window.

*Under equal function-evaluation budgets on CPU, which gradient-free optimizer (Nelder-Mead, Powell, CMA-ES) gives the best trade-off between final objective quality, runtime, and success rate on non-convex benchmark functions?*

Hypotheses:
statement: CMA-ES achieves lower mean best objective value than both Nelder-Mead and Powell on at least 2 of 3 benchmark functions, averaged over >=10 random starts with the same evaluation budget., statement: Powell has lower median wall-clock runtime than CMA-ES on at least 2 of 3 benchmark functions while finishing within the same function-evaluation cap., statement: On the multimodal Rastrigin function, Nelder-Mead attains a lower success rate (fraction of runs reaching f(x) <= 1e-2) than CMA-ES by at least 0.20 absolute.

Experiment design:
research_question: Under equal evaluation budgets, how do Nelder-Mead, Powell, and CMA-ES compare on objective quality, runtime, and success probability for non-convex synthetic objectives?
conditions: name: nelder_mead
description: scipy.optimize.minimize with method='Nelder-Mead', random initial point per run, maxfev budget enforced., name: powell
description: scipy.optimize.minimize with method='Powell', same initialization protocol and maxfev cap., name: cma_es
description: CMA-ES implementation (lightweight numpy/scipy variant) with population updates under the same total function-evaluation budget., name: random_search_baseline
description: Uniform random search within bounded domain using the same number of objective evaluations as other methods.
baselines: random_search_baseline as a budget-matched non-adaptive baseline
metrics: name: primary_metric
direction: minimize
description: Mean best objective value found at termination (lower is better), aggregated over random starts., name: median_runtime_sec
direction: minimize
description: Median wall-clock runtime per run in seconds for each (method, function)., name: success_rate_eps
direction: maximize
description: Fraction of runs reaching objective value <= 1e-2 by termination.
datasets: name: rastrigin_10d
source: synthetic numpy implementation of Rastrigin function in 10 dimensions, name: rosenbrock_10d
source: synthetic numpy implementation of Rosenbrock function in 10 dimensions, name: ackley_10d
source: synthetic numpy implementation of Ackley function in 10 dimensions
compute_requirements: gpu_required: no
estimated_wall_clock_sec: 480

Requirements:

Scoring rubric:
requirements: A credible experiment comparing Nelder-Mead, Powell, and CMA-ES on non-convex benchmark functions: the optimizers are implemented under matched evaluation budgets, execution covers multiple benchmark functions with repeated random starts, and results address H1/H2/H3 directionally.
judging_note: Score on scientific substance and directional correctness of evidence, not on exact threshold satisfaction. If the submission uses different optimizer names or slightly different benchmark functions but addresses the same scientific question, credit it accordingly.
sub_tasks: requirements: The optimization methods and benchmark setup are implemented in a way that supports a fair comparison.
sub_tasks: requirements: The submission implements distinct optimizer code paths for a local-simplex method (e.g., Nelder-Mead), a direction-set method (e.g., Powell), and an evolution-strategy method (e.g., CMA-ES or a well-motivated substitute such as a random-search baseline), rather than aliases of a single optimizer.
task_category: Code Development
finegrained_task_category: Method Implementation, requirements: The submission defines multiple non-convex benchmark functions (e.g., Rastrigin, Rosenbrock, Ackley at a moderate dimension such as 10D) with bounded domains and deterministic objective evaluation.
task_category: Code Development
finegrained_task_category: Dataset and Model Acquisition, requirements: A shared function-evaluation budget (maxfev or equivalent) is enforced across optimizers so comparisons are budget-matched.
task_category: Code Development
finegrained_task_category: Experimental Setup, requirements: Execution uses repeated random starts and produces comparable benchmark metrics.
sub_tasks: requirements: Execution uses multiple random starts per (optimizer, function) cell across the evaluated benchmark functions. More starts are better for reducing variance, but an honest small-repetition run with reported dispersion is preferable to a single start.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: Execution produces a machine-readable metrics artifact with numeric best-objective-value (the primary optimization metric), a wall-clock runtime measure (e.g., median runtime), and a success-rate-style metric (fraction of runs reaching a threshold f(x) ≤ ε) per optimizer and function.
task_category: Code Execution
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The reported results address all three hypotheses and include a clear narrative.
sub_tasks: requirements: The submission compares mean best-objective-values across the optimizers for each function and conveys whether CMA-ES (or its stand-in) tends to outperform the local methods on the multimodal functions — judge directionally against H1.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares runtime between Powell and CMA-ES across the evaluated functions and conveys whether Powell tends to finish meaningfully faster under the matched budget.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation, requirements: The submission compares success-rate on a hard multimodal function (e.g., Rastrigin) between a local method (e.g., Nelder-Mead) and CMA-ES and conveys whether CMA-ES has a clearly higher success rate.
task_category: Result Analysis
finegrained_task_category: Evaluation, Metrics & Benchmarking, requirements: The README or writeup describes the benchmark setup and metrics, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations such as budget size, dimensionality, CMA-ES implementation simplifications, and seed variance. No strict word-count requirement.
task_category: Result Analysis
finegrained_task_category: Logging, Analysis & Presentation