Package: pypoolparty
A python package for job pools (as inmultiprocessing.Pool()) which makes
use of distributed compute clusters.Thepypoolpartyprovides aPool()with amap()function which aims
to be a drop-in-replacement forbuiltins’map(), andmultiprocessing.Pool()’smap(). The idea is to allow the user to always fall back to these builtin pools and map-functions in case a distributed compute cluster is not available.This package respects the concept of ‘fair share’ what is commonly found
in scientific environments, but is not common in commercial environments.
Here, fair share means that compute resources are only requested when they
are needed. Compute resources are not requested to just idle and wait for
the user to submit jobs.A consequence of this fair sharing is, that this package expects your jobs
to randomly die in conflicts for resources with jobs submitted by other users,
such as conflicts for limited disk space on temporary drives. If your jobs run
into error states, they will be resubmitted until a predefined limit is
reached.InstallingpipinstallpypoolpartyBasic Usageimportpypoolpartyasppppool=ppp.slurm.Pool()results=pool.map(sum,[[1,2],[2,3],[4,5],])Currently, there isppp.slurm.Pool()andppp.sun_grid_engine.Pool().AlternativesWhen you do not share resources with other users, when you do not need to respect fair share, and when you have some administrative power you might want to use one of these:Daskhas ajob_queuewhich also supports other flavors such as PBS, SLURM.pyABC.sgehas apool.map()very much like the one in this package.ipyparallelQueue FlavorsSLURM, version 22.05.6Sun Grid Engine (SGE), version 8.1.9Inner Workingsmap()makes awork_dirbecause the mapping and reducing takes place in the filesystem. You can setwork_dirmanually to make sure both the worker nodes and the process node can reach it.map()serializes yourtasksusingpickleinto separate files inwork_dir/{ichunk:09d}.pkl.map()reads all environment variables in its process.map()creates the worker-node script inwork_dir/worker_node_script.py. It contains and exports the process’ environment variables into the batch job’s context. It reads the chunk of tasks inwork_dir/{ichunk:09d}.pkl, imports and runs yourfunc(task), and finally writes the result back towork_dir/{ichunk:09d}.pkl.out.map()submits queue jobs. Thestdoutandstderrof the tasks are written towork_dir/{ichunk:09d}.pkl.oandwork_dir/{ichunk:09d}.pkl.erespectively. By default,shutil.which("python")is used to process the worker-node-script.When all queue jobs are submitted,map()monitors their progress. In case a queue-job runs into an error-state, the job will be deleted and resubmitted until a maximum number of resubmissions is reached.When no more queue jobs are running or pending,map()will reduce the results fromwork_dir/{ichunk:09d}.pkl.out.In case of non-zerostderrin any task, a missing result, or on the user’s request, thework_dirwill be kept for inspection. Otherwise its removed.Environment VariablesAll the user’s environment variables in the process wheremap()is called
will be exported in the queue job’s context.The worker-node script explicitly sets the environment variables.
This package does not rely on the batch system’s ability (slurm/sge)
to do so.Wordingtaskis a valid input tofunc. Thetasksare the actual payload to be processed.iterableis an iterable (list) oftasks. It is the naming adopted frommultiprocessing.Pool.map.itaskis the index of ataskiniterable.chunkis a chunk oftaskswhich is processed on a worker-node in serial.ichunkis the index of a chunk. It is used to create the chunks’s filenames such aswork_dir/{ichunk:09d}.pkl.queue-jobis what we submit into the queue. Each queue-job processes the tasks in a single chunk in series.jobnameorjob["name"]is assigned to a queue job by ourmap(). It is composed of ourmap()’s session-id, andichunk. E.g."q"%Y-%m-%dT%H:%M:%S"#{ichunk:09d}"Testingpytest-s.dummy queueTo test ourmap()we provide a dummyqsub,qstat, andqdelfor the sun-grid-engine.
These are individualpythonscripts which all act on a common state file
intests/resources/dummy_queue_state.jsonin order to fake the
sun-grid-engine’s queue.dummy_qsub.pyonly appends queue jobs to the list of pending jobs in the state-file.dummy_qdel.pyonly removes queue jobs from the state-file.dummy_qstat.pydoes move the queue jobs from the pending to the running list, and does trigger the actual processing of the jobs. Each timedummy_qstat.pyis called it performs a single action on the state file. So it must be called multiple times to process all jobs. It can intentionally bring jobs into the error-state when this is set in the state-file.Before running the dummy queue, its state file must be initialized:frompypoolpartyimportsun_grid_enginesun_grid_engine.testing.init_queue_state(path="tests/resources/dummy_queue_state.json")When testing ourmap()you set its argumentsqsub_path,qdel_path,
andqstat_pathto point to the dummy queue.Seetests/test_full_chain_with_dummy_qsub.py.Because of the global state file, only one instance of dummy_queue must run
at a time.