Package: xqute
xquteA job management system for pythonFeaturesWritten in asyncPlugin systemScheduler adaptorJob retrying/pipeline halting when failedInstallationpip install xquteA toy exampleimportasynciofromxquteimportXquteasyncdefmain():# 3 jobs allowed to run at the same timexqute=Xqute(scheduler_forks=3)for_inrange(10):awaitxqute.put('sleep 1')awaitxqute.run_until_complete()if__name__=='__main__':asyncio.run(main())APIhttps://pwwang.github.io/xqute/UsageXqute objectAn xqute is initialized by:xqute=Xqute(...)Available arguments are:scheduler: The scheduler class or nameplugins: The plugins to enable/disable for this sessionjob_metadir: The job meta directory (Default:./.xqute/)job_error_strategy: The strategy when there is error happenedjob_num_retries: Max number of retries when job_error_strategy is retryjob_submission_batch: The number of consumers to submit jobsscheduler_forks: Max number of job forks**scheduler_opts: Additional keyword arguments for schedulerNote that the producer must be initialized in an event loop.To push a job into the queue:awaitxqute.put(['echo',1])Using SGE schedulerxqute=Xqute('sge',scheduler_forks=100,qsub='path to qsub',qdel='path to qdel',qstat='path to qstat',sge_q='1-day',# or qsub_q='1-day'...)Keyword-arguments with names starting withsge_will be interpreted asqsuboptions.listortupleoption values will be expanded. For example:sge_l=['h_vmem=2G', 'gpu=1']will be expanded in wrapped script like this:# ...#$ -l h_vmem=2G#$ -l gpu=1# ...Using Slurm schedulerxqute=Xqute('slurm',scheduler_forks=100,sbatch='path to sbatch',scancel='path to scancel',squeue='path to squeue',sbatch_partition='1-day',# or slurm_partition='1-day'sbatch_time='01:00:00',...)Using ssh schedulerxqute=Xqute('ssh',scheduler_forks=100,ssh='path to ssh',ssh_servers={"server1":{"user":...,"port":22,"keyfile":...,# How long to keep the ssh connection alive"ctrl_persist":600,# Where to store the control socket"ctrl_dir":"/tmp",},...}...)SSH servers must share the same filesystem and using keyfile authentication.PluginsTo write a plugin forxqute, you will need to implement the following hooks:on_init(scheduler): Right after scheduler object is initializedon_shutdown(scheduler, sig): When scheduler is shutting downon_job_init(scheduler, job): When the job is initializedon_job_queued(scheduler, job): When the job is queuedon_job_submitted(scheduler, job): When the job is submittedon_job_started(scheduler, job): When the job is started (when status changed to running)on_job_polling(scheduler, job): When job status is being polledon_job_killing(scheduler, job): When the job is being killedon_job_killed(scheduler, job): When the job is killedon_job_failed(scheduler, job): When the job is failedon_job_succeeded(scheduler, job): When the job is succeededNote that all hooks are corotines excepton_initandon_shutdown, that means you should also implement them as corotines (sync implementations are allowed but will be warned).To implement a hook, you have to fetch the plugin manager:fromsimplugimportSimplugpm=Simplug('xqute')# orfromxquteimportsimplugaspmand then use the decoratorpm.impl:@pm.impldefon_init(scheduler):...Implementing a schedulerCurrently there are only 2 builtin schedulers:localandsge.One can implement a scheduler by subclassing theSchedulerabstract class. There are three abstract methods that have to be implemented in the subclass:fromxquteimportScheduerclassMyScheduler(Scheduler):name='my'job_class:MyJobasyncdefsubmit_job(self,job):"""How to submit a job, return a unique id in the scheduler system(the pid for local scheduler for example)"""asyncdefkill_job(self,job):"""How to kill a job"""asyncdefjob_is_running(self,job):"""Check if a job is running"""As you may see, we may also need to implement a job class beforeMyScheduler. The only abstract method to be implemented iswrap_cmd:fromxquteimportJobclassMyJob(Job):asyncdefwrap_cmd(self,scheduler):...You have to use the trap command in the wrapped script to update job status, return code and clear the job id file.