Package: minionize
RationaleYou wrote a programa.outwith some parametersYou need to explore the space of parameters (or at least a large subset of it)Minionizeis a solution to spawn a legion ofa.outin a massively
parallel manner. Byminionizingyour program, yourminionswill take
their inputs from various sources (e.g filesystem, pub/sub) and run it. Also
inputs can be acked or redelivered to another minions.You can also think aboutminionizeas an equivalent to the “| xargs”
shell construction but where:thepipe |can transmit your data from many different sources (not
only from the stdout of the previous process)thexargscan transform the received data before sending them to your
program.That being said,Minionizeis a good choice for yourbesteffort and
idempotentcomputations on scientific testbeds (Igrida, Grid’5000) but also if you need
a quick way to turn you binary into a micro-service (e.g for Kubernetes).Minionizeprovides some observability features out-of-the box:
it exposes an api to retrieve the stats and a basic cli to display some of
them (to be honest, I dream of anmtop:htopfor minions). But
currently it looks more or less like this:How does it workA classical pattern to do the above is to apply the master/worker pattern
where a master gives tasks to workers. Workers repeatedly fetch a new task
from a queue , run it and report back to the master its status (e.g success,
failure).Minionizeapplies somehow this pattern but ismasterlessout-of-the box. Indeed modern queue implementations expose APIs to
acknowledge/requeue messages.Currently we support:For thepipe |part:execobased queue: the queue is stored in a shared file system in your cluster (no external process involved)Google pub/subbased queue: the queue is hosted by GoogleApache pulsar, a pub/sub system you can self-hostfile, take inputs from a regular file (e.g stdin)For thexargspart:processes: launch you program isolated in a process upon reception.functions: launch a python function upon reception.docker: launch a docker container upon reception.Some examplesSimplest use caseIn this case the received params are appended to the
minionized program. If you need more control on the params you’ll need to
write your own Callback (see below).withExecoengine:# Install the execo minionizerpipinstallminionize[execo]# Create the queue of params
# You'll have to run this prior to launching your minions (adapt to
# your need / make a regular script)$)python-c"from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(10), save_sweeps=True)"# start your minions$)MINION_ENGINE=execominionizeechohellohello0hello1hello2hello3hello4hello5hello6hello7hello8hello9NoteIn other words theminionizewrapper lets you populate the queue
with strings representing the parameter of your command lineRecord some stats: you need to setup aReporterto report your stats.# Install the execo minionizerpipinstallminionize[execo]# Create the queue of params
# You'll have to run this prior to launching your minions (adapt to
# your need / make a regular script)$)python-c"from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(10), save_sweeps=True)"# start your minionsMINION_ENGINE=execoMINION_REPORTER=jsonminionizesleep# read the stats (while running or no)MINION_REPORTER=jsonminion-statusOn a OAR cluster (Igrida/Grid5000):Generate the queue for example with Execopython-c"from execo_engine.sweep import ParamSweeper; ParamSweeper('sweeps', sweeps=range(1000), save_sweeps=True)"Create your oar scan script:#!/usr/bin/env bash#OAR -n kpd
#OAR -l nodes=1,walltime=1:0:0
#OAR -t besteffort
#OAR -t idempotent# oarsub --array 10 -S ./oar.shset-euxpipinstallminionizeminionizeecho"hello from$OAR_JOB_ID"Start your minionsecho"MINION_ENGINE=execo">.envoarsub--array10-S./oar.shNote.envfile is read when minionizing starts so the scan script can
remain the same whatever engine is used.Example of output:$)catOAR.1287856.stdout[...]hellofrom1287856135hellofrom1287856139hellofrom1287856143hellofrom1287856147hellofrom1287856151hellofrom1287856155hellofrom1287856159hellofrom1287856163hellofrom1287856167[...]NoteAs expected params have been distributed to different minionsCustom CallbacksThe params sent to your program can be anything (e.g a python dict). In
some cases (many actually), you’ll need to transform these params to
something that you program can understand.So you’ll need to tell
minionize how to minionize. This is achieved using specific callbacks.The easiest way to write a custom callbacks is to inherit fromProcessCallbackorFuncCallback. With these Callbacks you don’t
have to worry about the acknowledgement logic.#   a.out is invoked like this: a.out --arg1 varg1 varg2#   but the queue holds json like object:#   {"arg1": varg11, "arg2": varg21}, {"arg1": varg12, "arg2": varg22} ...# we can write a custom ProcessCallback which overrides the to_cmd methodclassMyProcessCallBack(ProcessCallback):defto_cmd(param:Param):returnf"a.out --arg1{param['arg1']}{param['arg2']}"m=minionizer(MyProcessCallback())m.run()#   you want to minionize a python function `my_function`#   but the queue holds json like object:#   {"arg1": varg11, "arg2": varg21}, {"arg1": varg12, "arg2": varg22} ...# we can use the FuncCallback for this purposedefmyfunc(...)# this is your functiondef_myfunc(param:Param)# this is the wrapper which invokes myfunc based on the paramsreturnmyfunc(param["arg1"],param["arg2"])m=minionizer(FuncCallback(_myfunc))m.run()Environment variablesMinionizeis configured using environment variables.
By default it reads a.envfile in the current directory but doesn’t
override existing system environment variables.Default values--------------------------------------------# which engine (queue implementation) to useMINION_ENGINE=execo# google, pulsar# ExecoEXECO_PERSISTENCE_DIR=sweeps# GoogleGOOGLE_PROJECT_ID=/mandatory/GOOGLE_TOPIC_ID=/mandatory/GOOGLE_SUBSCRIPTIOn=/mandatory/GOOGLE_APPLICATION_CREDENTIALS=/mandatory/GOOGLE_DECODER=identity# PulsarPULSAR_CONNECTION=pulsar://localhost:6650PULSAR_TOPIC=/mandatory/PULSAR_DECODER=identity---------------------------------------------# Stat reporterMINION_REPORTER=null# json, stdout# JsonREPORTER_JSON_DIRECTORY=minion-reportRoadmapEasy integration as docker entrypointMinionize python function (e.g @minionize decorator)Support new queues (Apache pulsar, Redis stream, RabbitMQ, Kakfa …)Support new abstractions to run container based application (docker, singularity…)Automatic encapsulation using a .minionize.ymlMinions statisticsKeep in touch (matthieu dot simonin at inria dot fr)