AHD-CMA

Adaptive Hybrid Dhole–CMA-ES Optimizer for LoRA-based Parameter- Efficient Fine-Tuning of Vision Transformers.

This repository accompanies the paper "AHD-CMA: An Adaptive Hybrid Dhole–CMA-ES Optimizer with Probe-then-Lock Phase Switching for Numerical Optimization and LoRA Hyperparameter Tuning of Vision Transformers" (under review, Soft Computing). It contains the full implementation, the benchmark + tuning harnesses, every figure/table generator, and the released result data.

Headline contributions:

• Probe-then-lock graceful-fallback controller — a hybrid runs a short pure-CMA-ES probe, commits to CMA-ES exploitation when the landscape proves locally quadratic, and otherwise releases a stagnation-triggered DOA-style hybrid burst. Presented as a transferable controller pattern, not only a single algorithm.
• AHD-CMA, the first concrete instantiation pairing CMA-ES with the Dhole Optimization Algorithm (DOA, 2025) under that controller.
• Reproducible evaluation on the official CEC-2022 suite (official shift + rotation + bias via opfunu, dims 10 and 20), plus a LoRA hyperparameter-tuning case study, with resumable JSON-per-run output and committed scripts that regenerate all figures and tables.

On the official rotated CEC-2022 suite AHD-CMA ranks 3rd of 16 at dim 10 and 4th of 16 at dim 20 (16 algorithms, 12 functions, 30 seeds), matching or outranking vanilla CMA-ES and trailing only the IPOP-/BIPOP-CMA-ES restart strategies. See docs/spec_deviations.md for the full audit trail of where the implementation diverges from the original proposal (notably, the chaotic Tent-map initialiser was found to be empirically inert and is not claimed as a contribution).

Results at a glance

Mean Friedman rank on the official CEC-2022 suite at dimension 20 (lower is better) — AHD-CMA matches/outranks vanilla CMA-ES and trails only the restart strategies:

Per-function ranking of the top algorithms, and the component ablation (blue = the variant beats the full algorithm on that function):

All figures are regenerated from the released result data with scripts/generate_paper_artifacts.py (see Data availability below).

Install

python3.11 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu121

Quick start

# Single official CEC-2022 run (function keys: F1_zakharov, F5_levy, ...)
python -m ahdcma.cli.run_benchmark \
    --algo ahdcma --func F5_levy --dim 20 --seed 0

# Single LoRA run (downloads the dataset + backbone the first time)
python -m ahdcma.cli.run_task \
    --algo ahdcma --task cifar100_vit --seed 0 \
    --pop 8 --gens 10 --num-steps 100

# Full CEC-2022 sweep — single-threaded shards across cores (~2-3 h)
bash scripts/run_cec_parallel.sh
# (or serial: python scripts/run_cec2022_full.py)

# Regenerate every figure from the sweep results
python scripts/generate_paper_artifacts.py \
    --cec-root outputs/runs/cec2022_rotated \
    --ablation-root outputs/runs/ablation_rotated \
    --out outputs/figures

See docs/experiments.md for the full reproduction recipe and docs/algorithm.md for the mathematical exposition.

Repository layout

configs/                   per-run YAML for algorithms and tasks
docs/                      algorithm description, experiment recipes,
                           changelog, spec deviations
src/ahdcma/                package source
  algorithms/              14 optimizer implementations + base class
  controller/              entropy / ruggedness / phase-switch
  fitness/                 CEC-2022 base functions and LoRA fine-tune
  search_space/            11-dim LoRA encoder + Tent-map init
  stats/                   Wilcoxon, Friedman, Cliff's delta
  viz/                     convergence + ablation + LaTeX tables
  cli/                     entry points (run_benchmark, run_task,
                           run_all, make_figures)
scripts/                   sweep launchers + ablation
tests/                     unit + gated integration tests
outputs/                   runs, logs, figures, tables (gitignored
                           where appropriate)

Data availability

All result files generated and analysed in the paper — 11,520 official CEC-2022 runs (dims 10 and 20), 1,800 ablation runs, 270 LoRA tuning runs, and 150 sensitivity runs as JSON records — are released as compressed archives under the Releases tab and permanently archived on Zenodo: https://doi.org/10.5281/zenodo.20587560. See data_release/README.md for the archive contents, schema, and the command to reproduce every figure from them. Everything is released under the MIT licence.

Citation

@article{tran2026ahdcma,
  title   = {{AHD-CMA}: An Adaptive Hybrid Dhole--{CMA-ES} Optimizer
             with Probe-then-Lock Phase Switching for Numerical
             Optimization and {LoRA} Hyperparameter Tuning of Vision
             Transformers},
  author  = {Tran, Huy Hoang Son},
  journal = {Soft Computing},
  year    = {2026},
  note    = {Under review}
}

If you use the released code or data, please also cite the software archive via the Zenodo DOI (see CITATION.cff).