Package: iohxplainer
IOHxplainereXplainable benchmarking using XAI methods for iterative optimization heuristics.IOHxplainer builds on top of theIOHpackage, including IOHexperimenter and IOHprofiler.InstallationEither install the package from source or via Pypi:pip install iohxplainerQuick StartImport the packagefromiohxplainerimportexplainerDefine thesearch spaceand the algorithm runner, basically the hyper-parameters and ranges you want to explore for a given optimization algorithm or modular algorithm. In this example we use the Differential Evolution implementation byscipyand we take a limited set of hyper-parameter options to explore.fromscipy.optimizeimportdifferential_evolutionfromConfigSpaceimportConfigurationSpaceimportnumpyasnpconfSpace=ConfigurationSpace({"strategy":["best1bin","best1exp","rand1exp","randtobest1exp"],"popsize":[1,2,5,10],})features=["strategy","popsize"]defrun_de(func,config,budget,dim,*args,**kwargs):bounds=[(-5,5)]*dim#define the boundariesresult=differential_evolution(func,bounds,strategy=config.get("strategy"),popsize=config.get("popsize"))Inside the algorithm runner (run_de) we define the boundaries of the to be optimized function. Since we will be using the BBOB benchmark suite for this example, we set the boundaries to -5,5 for each dimension of the problem.Next we create theexplainerobject and run the experiments.de_explainer=explainer(run_de,confSpace,algname="Differential Evolution",dims=[2],# we test in 2D for this example.fids=np.arange(1,25),# we use all 24 BBOB functions.iids=[1,2,3,4,5],# we use the first 5 instances for each BBOB function.reps=3,# we repeat each run 3 times with different random seeds.sampling_method="grid",# can also be set to random, since we have a few options we can make a full enumeration of the space.grid_steps_dict={},# only needed if we have continuous parameters.sample_size=None,# only used with random sampling methodbudget=10000,# evaluation budget of an optimization run.seed=1,# starting random seed for reproducability.verbose=True,)#we start running the experiments in paralell and store intermediate results in a csv (to allow a restart when crashing)de_explainer.run(paralell=True,start_index=0,checkpoint_file="checkpoint.csv")#store the final results as pkl file.de_explainer.save_results("de.pkl")Finally we can analyze the run data and make various plots and reports.importosos.mkdir("de_report")de_explainer.to_latex_report(filename="de_report",img_dir="de_report/")Also see thepython notebook filefor additional demo material.Experimental setupAll experiments and setup from the scientific paper can be found in the (experiments)[experiments/] folder.The Modular CMA and Modular DE setup are specified in theconfig.pyfile.Reproducing the experimentsSteps to reproduce the experiments from the paper.
Make sure you checkout the ModularCMAES framework from Github and install the cpp version.
Seehttps://github.com/IOHprofiler/ModularCMAESfor installation instructions.The experiments are run on two modular frameworks, Modular DE and Modular CMAES. Both can be easily installedpoetry add modde modcma.Run all Modular DE or Modular CMA configurations using the(de|cma_es)_run-configurations.pyfile, writes a pkl file as result.
(This step takes a few days on a supercomputer with 120 cores). The processed results of this step and the step 2 can also be downloaded (since it takes roughly a month to run on a CPU cluster): (cma_final_processed.pkl)[https://www.dropbox.com/scl/fi/ry9b1nnn7681o3b08o073/cma_final_processed.pkl?rlkey=zi9kjjs8t870ldzx9iw87fpm9&dl=0] and (de_final_processed.pkl)[https://www.dropbox.com/scl/fi/f46q2tuhylupm7vgth948/de_final_processed.pkl?rlkey=j87etm66ilvglue0l35mvw8n7&dl=0].Pre-process the pickle files with(de|cma_es)_process_pkl.py.Analyse the performance data of all configurations using IOH-Xplainer(de|cma_es)_analyse.py.Compare the two frameworks usingcompare_de_cma.py. Writes the result as latex file (compare-new.tex).Perform automated algorithm configuration experiment using(de|cma_es)_AAC-notebook.ipynbfiles.Setting up the dev environmentCheckout this code.Make surepipx(https://github.com/pypa/pipx) is installed with python 3.8+Install Poetry with pipxpipx install poetryInstall dependenciespoetry installRun testspoetry run pytestCite usTBA