Package: orqal
Orqal for [ORQ]chestration of [AL]gorithms is a simple batch scheduler for docker cluster which can be used remotly and without overhead in scientific experiment.📐 DesignOverviewOrqal claim to be and stay as simple as possible.We use the docker api to manage docker nodes, which means that there isno configurationon node except open the api port.HTTP/REST APIenable to schedule jobs and retrieve data.Adashboardis provided to monitor load average per nodes, jobs scheduling and redoc api.WrapperThe glue between docker image and orqal need to be implemented in python with the ArbstractWorker class in order to :get command to executeset the result if necessaryHere is a simple example using radare2classRabin2(AbstractWorker):docker_url="radare/radare2"volumes={'/database':{'bind':'/database','mode':'ro'}}threads=1memory_in_gb=1defget_cmd(self,params):return"rabin2 -I%s"%self.job.inputdefset_result(self,job):r={l.split()[0].replace('.','_'):l.split()[1]forlinjob.stdoutiflen(l.split())==2}job.set_result(r)Installpip3 install orqalRunStart the web worker :orqal-workerStart the web interface :orqal-webNB: In a production mode, use gunicorn, in orqal dir :gunicorn3 web:app --bind 0.0.0.0:5001 --worker-class aiohttp.GunicornWebWorker --workers 8 --timeout 120userConfigurationIn order to establish a DB connection, process will search the following environment variable :ORQAL_MONGO_URIBy default :  'mongodb://localhost/'Other settings will be loaded from the conf collection, using the request : active=TrueHere is a example :NB : The first run will initiate it if necessary{"_id":ObjectId("5d1a78eca307b20dd5a660ce"),"active":true,"mongourl":"mongodb://localhost/","docker_hosts":["192.168.100.51:2376",....."192.168.100.64:2376"],"docker_api_version":"1.37","mongo_replicaset":"madlabReplSet","registry_auth_config":{"password":"65sX2-9sSXSp-hs-XeZ8","username":"test"},"services":"/home/madlab/services.py","nb_disp_jobs":30,"contact":"orqal@example.com","jobs_dir":"/scratch/jobs"}ServicesThe python module is defined in the configuration (see services above)Here is a example :importjsonimportloggingimportconfimportosfromorqal.abstractimportAbstractWrapperclassTestProd(AbstractWrapper):docker_url="madlab:5000/test_module"defget_cmd(self,params):return"python3 simple_job.py%s%s%s"%(params['echo'],params['time'],params['exit_code'])defset_result(self,job):job.set_result("My results")classAngrExtraction(AbstractWrapper):docker_url="madlab:5000/scdg/madlab-v2"threads=1memory_in_gb=10create_dir=Truedefget_cmd(self,params):return"python /code/src/interfaces/cli.py%sparams.json -o calls.json"%self.job.inputdefset_result(self,job):returnos.path.join(self.job.wd,"calls.json")DB tuning : log collection in MongoProcess use a mongo logging handler, in order to provide it in the web interface, you can customize parameters like that :use orqal
db.createCollection('log', {capped:true, size:10000000}) 
db.log.createIndex( { "time": 1 }, { expireAfterSeconds: 86400 } )Cache :use orqal
db.jobs.createIndex( { input: 1 } )
db.jobs.createIndex( { params: 1 } )
db.jobs.createIndex( { app: 1 } )Distributionpython setup.py bdist
   python3 setup.py sdist
   twine upload dist/*LicenceGNU AFFERO GENERAL PUBLIC LICENSEhttps://www.gnu.org/licenses/agpl-3.0.txtCreditsDashboard template :https://github.com/puikinsh/sufee-admin-dashboardFont :https://fonts.google.com/specimen/RighteousFAQHow to open api port on docker ?Add in file/etc/systemd/system/docker.service.d/override.conf[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd://  -H tcp://0.0.0.0:2376 -s overlay --insecure-registry orqal:5000Then flush changes by executing :systemctl daemon-reloadverify that the configuration has been loaded:systemctl show --property=ExecStart dockerrestart docker:systemctl restart dockerClean old jobsAdd a index (7 days here):db.jobs.createIndex( { "ctime": 1 }, { expireAfterSeconds: 604800 } )And call http:///api/clean/old periodically in order to delete old job directories.