Package: dibb
DiBBDistributing Black-Box optimizationTL;DRDiBB generates and runs a Partially-Separable, parallelized and distributed version of your favorite Black Box optimization algorithm -- including Evolutionary Algorithms such as Evolution Strategies for continuous optimization.# also installs CMA-ES, a solid choice for the underlying BBOpipinstalldibb[cma]`# PythonfromdibbimportDiBB# If a Ray cluster with 3 machines is already running, the computation will be# distributed as 2 block workers (5 dims each) and 1 head node# If no Ray cluster is running, the three processes will spawn locallydibb=DiBB(ndims=10,nblocks=2,fit_fn=lambdax:sum(x**2),verbose=True)dibb.run(ngens=10)Paper + bibtex are at the bottom of this file, or check outGiuse's page.Powerusers:don't miss out on theAdvanced Usagesection below.InstallationYou can install DiBB from Pypi usingpip install dibb[cma]ordibb[lmmaes], which installs both DiBB and a solid underlying optimizer (CMA-ESorLM-MA-ESrespectively). Runningpip install dibb[all]installs all currently available optimizers.If you justpip install dibb, it will install only DiBB (and Parameter Guessing); you can then install separately an optimizer of your choice, just make sure thatthere is already a wrapper available for it-- if not,writing your ownis very easy.To contribute, you can also clone the repo from GitHubgit clone https://github.com/giuse/dibbthen install it locally withpip install -e <path_to>/dibb.Local usageDiBB is compatible with several traditional workflows from the larger family of Black Box Optimization methods. Here's an example:# Let's define a simple function to optimize:importnumpyasnpdefsphere(coords):np.dot(coords,coords)# You can use the classic `fmin`, if you needfromdibbimportfminfmin(sphere,np.random.rand(10))# ...but that is not particularly interesting# Let's fire up a couple of *blocks* instead:fromdibbimportDiBBdibb=DiBB(ndims=10,nblocks=2,fit_fn=sphere,verbose=True)dibb.run(ngens=100)When launching DiBB using the last command, you should notice the overhead of Ray starting up with multiple workers to handle the blocks. It is just a few seconds, and already allows you to play with dimensionalityndimsmuch larger than you could with CMA-ES alone.
The algorithm used is more correctly the partially-separable version of CMA created by DiBB:PS-CMA-ES.Check out the paperfor more details.Cluster usageTo enable the full potential of DiBB you should get your hands on a few machines (one per each block, plus one for the head process), then:pip install rayon each of themSet up basic SSH key-pair authentication (here's aquick script)Mark down the IP addressesCustomize a copy ofray_cluster_config.yaml(instructions inside)Initialize the cluster by running (locally)ray up ray_cluster_config.yamlNow you can run the same code again, only this time it will find and utilize your new cluster rather than launching local processes. Notice the overhead is very small.# With a Ray cluster with 3 machines already running, this code will now# run two blocks on two machines, plus the head node on a third machinefromdibbimportDiBBdibb=DiBB(ndims=10,nblocks=2,fit_fn=sphere,verbose=True)dibb.run(ngens=100)A more flexible way to call DiBB with multiple options is to use adict(checkDiBB.defaultsfor an example):fromdibbimportDiBBdibb_config={'ndims':10,'nblocks':2,'fit_fn':sphere,'verbose':True,'stopping_conditions':{'ngens':100},}dibb=DiBB(**dibb_config).run()You can find the complete list of accepted parameters, their descriptions and default values, inDiBB's main fileNeuroevolution example in 5 minutessNeuroevolutionmeans training neural networks using evolutionary computation algorithms. This is not a supervised learning technique, so you need no labels and no differentiability. In a RL control problem for example, you can train a neural network to use as the policy (Direct Policy Search), and entirely ditch value functions, Markov chains and the whole classic RL framework.First you will need to install the requirements (besides DiBB of course): 1. an optimizer, 2. a neural network and 3. a RL control environment.pipinstall"dibb[cma] tinynet gym[classic_control]"(the quotes are here to escape the square parenthesis forzsh, which is currently the default shell on Mac)Then copy+paste the example below to a.pyfile and run it.
It should not take long, even on your local machine -- or you can launch a cluster of 3 machines first (usingray_cluster_config.yaml): the example below will run on the cluster with no further changes.# INSTALL REQUIREMENTS FIRST: dibb (with optimizer), neural network, RL environment:# $ pip install "dibb[cma] tinynet gym[classic_control]"importnumpyasnpimportrayimporttinynet# https://github.com/giuse/tinynet/importgym# https://www.gymlibrary.ml/environments/classic_control/importwarnings# silence some gym warningswarnings.filterwarnings(action='ignore',category=UserWarning,module='gym')fromdibbimportDiBB# Set up the environment and networkenv=gym.make("CartPole-v1")nactions=env.action_space.nninputs=env.reset().size# Just 2 neurons (for the Cartpole only has 2 actions: left and right)# with linear activation `f(x)=x` are already enoughnet=tinynet.FFNN([ninputs,nactions],act_fn=lambdax:x)# The fitness function accumulates the episodic reward (basic gym gameplay)defplay_gym(ind,render=False):obs=env.reset(seed=1)# fix random seed: faster training but less generalizationnet.set_weights(ind)# set the weights into the networkscore=0done=Falsewhilenotdone:ifrender:env.render()# you can watch it play!action=net.activate(obs).argmax()# pick action of neuron with highest actobs,rew,done,info=env.step(action)# With NE we ignore the availability of per-action reward (it's rarely# available in nature anyway), and just accumulate it over the episodescore+=rewreturnscore###### THIS IS THE ACTUAL DIBB-SPECIFIC CODE ####### Call `DiBB.defaults` for the full list, check `dibb/dibb.py` for descriptionsdibb_config={'fit_fn':play_gym,'minim_task':False,# IMPORTANT: in this task we want to _maximize_ the score!'ndims':net.nweights,'nblocks':net.noutputs,# Let's use a block for each output neuron (2 here)'optimizer':'default',# CMA or LMMAES if installed, else Parameter Guessing# 'optimizer_options' : {'popsize' : 50} # Ray manages parallel fitness evals}dibb=DiBB(**dibb_config).run(ngens=15)# Cartpole is not a challenge for DPS/NE#################################################### Watch the best individual play!best_fit=play_gym(dibb.best_ind,render=True)print("Best fitness:",best_fit)# You can even resume the run for a few more generations if needed:# print("Resume training for 15 more generations")# dibb.run(ngens=15)# print("Best fitness:", play_gym(dibb.best_ind, render=True))# Keep the console open at the end of the run, so you can play more with DiBB!print("\n\nDropping into an interactive console now, feel free to inspect the `dibb` object at your leisure. For example, try running `dibb.defaults` to see all options and their default values; `dibb.cfg` for the values actually used for this run; `dibb.result` for a dict of the final mean and best individual found; `dibb.comm_dict` to see the communication dictionary with the BlockWorkers.\n")importos;os.environ['PYTHONINSPECT']='TRUE'GlossaryDiBB:Distributed Black Box (optimization)Block:subset of parameters assumed to be correlated (intra-block); parameters in different blocks are assumed separable (inter-block). Notice that in most applications both full correlation and full separability between parameters are very unlikely. With DiBB you can better approximate real characteristics, and distribute the computation of blocks to different, dedicated machines as a bonus. Check the paper to know more.Optimizer:the underlying black box optimization algorithm that will be used on a block level. Notice that using an optimizer with DiBB automatically generates a partially-separable version of that algorithm: for example, using CMA-ES with DiBB you are actually running PS-CMA-ES.Block Worker (BW):the class, process and Ray actor that runs the base optimizer on one block. They are distributed to their own dedicated machines when a cluster is available. BWs always run asynchronously from each other, leveraging the assumption of block separability.Fitness (function):the objective function for the optimization. In classical Reinforcement Learning, the term "reward" indicates a reinforcement available per each interaction (denoted by states and actions). However, this is commonly not available in real-world processes. Black-box optimization instead uses a "fitness" evaluation, which is a score of proficiency of one solution (as it is a multi-agent setup) on the task. The quickest way to build a fitness from an environment coded with classic RL in mind is to accumulate the reward throughout the episode, though customizing a more proper definition of "fitness" beyond fixed extrinsic reward schemes can often improve performance quite drastically -- so keep that in mind.Trial:one evaluation of one individual on the task, running the fitness function. If the environment fitness is non-deterministic (e.g. random initial conditions), you can provide a better estimation of the individual's performance by averaging across multiple trials. DiBB has an option to do this automatically.Fitness Evaluator (FE):each BW can spawn a pool of FE, sized to the machine's computational resources (e.g. number of CPUs or GPUs). All fitness computation then runs through the FEs, as an effective way to manage (potentially complex and external) available resources. For example: load-scaling when the fitness function is resource intensive, such as a robotic control task running on a separate physics simulator.Random seed:setting a random seed for the (pseudo-)random number generator enforces reproducibility of the whole run. If tackling a non-deterministic problem usingntrials>1, each trial sets a separate seed out of a sequence ([rseed, rseed+1, rseed+2, ... , rseed+ntrials-1]).Instantiation hierarchyThe cluster, if available, should be launched from a local machine, and include at leastnblocks+1machines (one for each BW + one for the head/DiBB)If no cluster is available, the run is local: all "machines" mentioned below become one and the same (the very local machine), hosting all computation.DiBB runs on the head machine, which also hosts the Rayobject store(hosting the shared data: check out theRay documentationfor further details)At initialization, the DiBB object instantiates a list of BlockWorkers (BWs), one for each parameters block, each assigned to a dedicated machine on the Ray cluster (if available)At initialization, each BW creates one instance of the underlying optimizer, which is dedicated to the subset of parameters of its assigned block -- more precisely, it will instantiation the correspondingopt_wrapper, ensuring a compatible interfaceEach BW can be set up to execute the fitness function either on its own (either sequentially or in parallel), or instantiate a pool of FitnessEvaluators (FE), typically as many as the available computational resources (e.g. CPU cores or GPUs) -- see Advanced Usage below for more infoIf using the FEs, these run on the same machine of the BW that instantiated them: at any given moment, the machine resources are either dedicated to fitness evaluation or to (underlying) BBO updateFurther distributing the fitness computation to extra available machines is a planned feature for a future releaseRun hierarchyDiBB instantiates the BWs, then remains at watch, executing the functionprint_fn(dibb)(if enabled) at custom regular intervalsIf chosen, each BW instantiates immediately an independent, machine-local pool of FEs to evaluate the individuals generated by its optimizerThe BW then immediately starts the generation loop of the underlying BBO, using a classic "ask&tell" interface:bbo.ask()needs to return a population of individuals, which are then evaluated on the fitness function by the BW (either locally or on the FEs)In case of multiple trials per individual, the BW aggregates the resulting episodic fitness scores using a customizable aggregation functionbbo.tell(inds, fits)then uses the individuals and fitness scores to execute one update step of the BBO algorithmThe BWs run asynchronously from each other, leveraging the assumption of inter-block separability. Potentially they could even run on machines with different performance, and the blocks can differ in size, though special care is then necessary to ensure load balanceThe BWs can individually decide to pause their execution, typically temporarily if convergence is achieved in one block (as themoving targetproblem should allow resuming the search shortly), or because of an inescapable termination criterion -- either way, this is communicated to the main DiBB process, which can itself request the BWs termination as needed, such as if they all reach termination.Advanced UsageThe complete list of available options to DiBB is available directly in the code to avoid duplication and simplify maintenance; you can check the top ofthe main DiBB file. You can ask directly the DiBB class (DiBB.defaults) or even the module (from dibb import options), but the source has more useful descriptions for each item.Theprintfnoption accepts a callable object (function, method, lambda, even a class implementing__call__()) which should take a DiBB object as parameter: this is then executed at regular intervals by the main node while waiting for the workers to finish (run report)Termination conditions can be passed more explicitly using thestopping_conditionsoption (e.g.DiBB(stopping_conditions={'nevals' : 10e5})Thehooksparameter accepts callables (either single or lists of them) which are executed before each generation (for the BW) or before each fitness evaluation (for the FE). This is often useful to customize the algorithm tracking and behavior, giving the user a direct "hook" into the algorithm's mechanicsFor reproducibility we usepipenvfor development. If you wish to contribute a pull request (or simply to run the tests), do apython3 -m pip install pipenvfollowed bypython3 -m pipenv syncto install the exact same environment we use for development (tip: don't usepipenv install, as it will update all libraries and edit thePipfile.lock). You can then run your virtualenv shell withpython3 -m pipenv shell, or run your code withpython3 -m pipenv run python your_experiment.pyTo run DiBB on a cluster:Download theRay cluster configuration templateOpen it in a text editor and follow instructions: you need to replace all placeholders (format looks like this:%replace_me%) with the actual data of your cluster (machine IPs, ray command, etc.)Save, close, and keep on your local machine, which should be able to reach the network hosting the cluster machinesInstall DiBB, Ray and all libraries for your experiment Ray on all machines of your cluster, including any external resource you need for your fitness evaluationLaunch the cluster by runningray up ray_cluster_config.yamlon your local machineLaunch your code by usingRay's job submission API. For a quick run (or to avoid problems e.g. with virtual environments through the job submission API) you can also simply connect to the head node withsshand launch your code from there. Done!You can run fitness computation either: 1. on the BlockWorker, sequentially; 2. on the BlockWorker, in parallel (the default); 3. on the FitnessEvaluator pool, ideal for complex fitnesses such as needing to launch external simulators on each machine in the clusterTo support complex fitness evaluation, especially on the FEs, and specifically to avoid the problems in pickling (serialization) complex fitness objects for distribution on the cluster: you can pass thefit_fnas astring, in which case it will be sent as a string and theneval()ed on the target machine. To fully leverage this, try the following:Create aFitnessclassIn its__init__, launch the simulator or connect to external resourcesDefine a__call__(self, ind)method to take an individual and return its fitness (on one run; use thentrialsoption for nondeterministic fitnesses)Pass to DiBB a string with a call to the constructor:DiBB(fit_fn="Fitness()")DiBB will send the string to the remote machines, which will then runfitness = eval("Fitness()"), thus creating locally the exact environment required for individual evaluationDebugging is best done followingRay's guidelines and tools. Particularly, be aware that Ray silently collects uncaught exceptions on the remote machines, and upon errors it does not flush the local stdout buffers to the main node.This means that if your code breaks a remote machine in the cluster, it will look like DiBB is failing silently and without support. There is nothing we can do about it (yet). Please be patient, either trust DiBB and simplify your fitness, or carefully debug and submit a fix as pull request if it's our fault. Thank you!Here's a snippet to help:# Classic printf debugging, add as many as neededimportinspectlineno=inspect.getframeinfo(inspect.currentframe()).linenoprint(f"\n\n\tline{lineno}of file{__file__}\n\n")then check the logs ("workers" here are machines in a Ray cluster):forlogin$(find/tmp/ray/session_latest/logs-name'worker*');doecho-e"\n\n--------$log--------\n"cat$logdoneAcknowledgementsThis work has been published at GECCO 2022, The 24th Genetic and Evolutionary Computation Conference.[Download the paper here], bibtex here:@inproceedings{cuccu2022gecco,author={Cuccu, Giuseppe and Rolshoven, Luca and Vorpe, Fabien and Cudr\'{e}-Mauroux, Philippe and Glasmachers, Tobias},title={{DiBB}: Distributing Black-Box Optimization},year={2022},isbn={9781450392372},publisher={Association for Computing Machinery},address={New York, NY, USA},doi={10.1145/3512290.3528764},booktitle={Proceedings of the Genetic and Evolutionary Computation Conference},pages={341–349},numpages={9},keywords={parallelization, neuroevolution, evolution strategies, distributed algorithms, black-box optimization},location={Boston, Massachusetts},series={GECCO '22},url={https://exascale.info/assets/pdf/cuccu2022gecco.pdf}}The experiment code to reproduce our COCO results isavailable here, created and maintained by Luca(@rolshoven).Since 2021 -- initially as part of his Master thesis, then out of his personal quest for excellence -- Luca Sven Rolshoven(@rolshoven)has contributed to this project with engaging discussions, reliable debugging, the original "hooks" feature,fundamental testing, and managing our cluster of 25 decommissioned office computers :)The git history of this repository has been wiped at publication for privacy concerns, but his contribution should not go unacknowledged nor underestimated. Thanks Luca!