DOI: 10.5281/zenodo.8109549
title: Experiment Data for A Per-Class Automated Algorithm Selection Framework for General Black-Box Optimisation

description: Raw experimental output, and data organised for analysis code (to be published separately) in paths ending with "_organised". All in IOHprofiler format.

Experimental settings:



Algorithms: All NGOpt choices for nevergrad version 0.5.0 (total 39 algorithms; nevergrad version 0.6.0 uses a subset of 34 of these algorithms for the same settings):



CMA,

ChainMetaModelPowell,

Cobyla,

MetaModel,

MetaModelOnePlusOne,

ParametrizedMetaModel(multivariate optimizer=CmaFmin2),

33 different ConfPortfolio versions




Data file specific content summary

BBOB TRAIN



data_seeds2.zip



Raw results

24 BBOB problems, 25 runs (each with a different seed) on instance 1 for every problem

Evaluation budget: 10000

Dimensions: 2, 3, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 100



data_seeds2_organised.zip



Data from data_seeds2.zip organised for use with analysis code



data_seeds2_bud_dep.zip



Raw results

F1 (sphere) BBOB problem, 25 runs (each with a different seed) on instance 1

Evaluation budget: 200

Dimensions: 100



data_seeds2_bud_dep_organised.zip



Data from data_seeds2_bud_dep.zip organised for use with analysis code



data_seeds2_ngopt.zip



Raw results

Algorithms: NGOpt39's choice per budget-dimension combination (on Nevergrad version 0.6.0)

24 BBOB problems, 25 runs (each with a different seed) on instance 1 for every problem

Evaluation budgets: 200, 300, 500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 6000, 7000, 8000, 9000, 10000

Dimensions: 2, 3, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 100



data_seeds2_ngopt_organised.zip



Data from data_seeds2_ngopt.zip organised for use with analysis code




BBOB TEST



data_seeds2_bbob_test.zip



Raw results

WARNING: Also contains preprocessed data from preprocessing during execution prior to a bugfix in the preprocessing code (raw data is valid and usable after reprocessing)

Algorithms: NGOpt39's choice, and the remaining top four algorithms per budget-dimension combination (i.e., five algorithms per budget-dimension combination)

24 BBOB problems, 1 run on per instance in 2-26 for every problem

Evaluation budgets: 200, 300, 500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 6000, 7000, 8000, 9000, 10000

Dimensions: 2, 3, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 100



data_seeds2_bbob_test_reprocessed.zip



Data from data_seeds2_bbob_test.zip organised and preprocessed data for use with analysis code; reprocessed after bugfix




MA-BBOB (TEST)



data_seeds2_ma_complete.zip



Raw results

WARNING: Also contains preprocessed data from preprocessing during execution prior to a bugfix in the preprocessing code (raw data is valid and usable after reprocessing)

Algorithms: NGOpt39's choice, and the remaining top four algorithms per budget-dimension combination (i.e., five algorithms per budget-dimension combination)

828 MA-BBOB problems, 1 run each (all composed combinations of instance 1 from the 24 BBOB problems)

Evaluation budgets: 200, 300, 500, 1000, 1500, 2000, 2500, 3000, 3500, 4000, 4500, 5000, 6000, 7000, 8000, 9000, 10000

Dimensions: 2, 3, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 90, 100



data_seeds2_ma_complete_reprocessed.zip



Data from data_seeds2_ma_complete.zip organised and preprocessed data for use with analysis code; reprocessed after bugfix




Package versions:



Performance data for all problem instances and algorithms was collected with nevergrad version 0.6.0.