Package: queue-map-reduce-sebastian-achim-mueller
Queues for batch-jobs distribute your compute-tasks over multiple machines in parallel. This pool maps your tasks onto a queue and reduces the results.importqueue_map_reduceasqmrpool=qmr.Pool()results=pool.map(sum,[[1,2],[2,3],[4,5],])A drop-in-replacement for builtins’map(), andmultiprocessing.Pool()’smap().RequirementsProgramsqsub,qstat, andqdelare required to submit, monitor, and delete queue-jobs.Yourfunc(task)must be part of an importable python module.Yourtasksand theirresultsmust be able to serialize using pickle.Both worker-nodes and process-node can read and write from and to a common path in the file-system.Queue flavorTested flavors are:Sun Grid Engine (SGE) 8.1.9FeaturesRespects fair-share, i.e. slots are only occupied when the compute is done.No spawning of additional threads. Neither on the process-node, nor on the worker-nodes.No need for databases or web-servers.Queue-jobs with error-state'E'can be deleted, and resubmitted until your predefined upper limit is reached.Can bundle tasks on worker-nodes to avoid start-up-overhead with many small tasks.AlternativesWhen you do not share resources with other users, and when you have some administrative power you might want to use one of these:Daskhas ajob_queuewhich also supports other flavors such as PBS, SLURM.pyABC.sgehas asge.map()very much like our one.ipyparallelInner workingsmap()makes awork_dirbecause the mapping and reducing takes place in the file-system. You can setwork_dirmanually to make sure both worker-nodes and process-node can reach it.map()serializes yourtasksusingpickleinto separate files inwork_dir/{ichunk:09d}.pkl.map()reads all environment-variables in its process.map()creates the worker-node-script inwork_dir/worker_node_script.py. It contains and exports the process’ environment-variables into the batch-job’s context. It reads the chunk of tasks inwork_dir/{ichunk:09d}.pkl, imports and runs yourfunc(task), and finally writes the result back towork_dir/{ichunk:09d}.pkl.out.map()submits queue-jobs. Thestdoutandstderrof the tasks are written towork_dir/{ichunk:09d}.pkl.oandwork_dir/{ichunk:09d}.pkl.erespectively. By default,shutil.which("python")is used to process the worker-node-script.When all queue-jobs are submitted,map()monitors their progress. In case a queue-job runs into an error-state ('E'by default) the job wFill be deleted and resubmitted until a maximum number of resubmissions is reached.When no more queue-jobs are running or pending,map()will reduce the results fromwork_dir/{ichunk:09d}.pkl.out.In case of non zerostderrin any task, a missing result, or on the user’s request, thework_dirwill be kept for inspection. Otherwise its removed.Wordingtaskis a valid input tofunc. Thetasksare the actual payload to be processed.iterableis an iterable (list) oftasks. It is the naming adopted frommultiprocessing.Pool.map.itaskis the index of ataskiniterable.chunkis a chunk oftaskswhich is processed on a worker-node in serial.ichunkis the index of a chunk. It is used to create the chunks’s filenames such aswork_dir/{ichunk:09d}.pkl.queue-jobis what we submitt into the queue. Each queue-job processes the tasks in a single chunk in series.JB_job_numberis assigned to a queue-job by the queue-system for its own book-keeping.JB_nameis assigned to a queue-job by ourmap(). It is composed of ourmap()’s session-id, andichunk. E.g."q"%Y-%m-%dT%H:%M:%S"#{ichunk:09d}"Environment VariablesAll the user’s environment-variables in the process wheremap()is called will be exported in the queue-job’s context.The worker-node-script sets the environment-variables. We do not useqsub’s argument-Vbecause on some clusters this will not set all variables. Apparently some administrators fear security issues when usingqsub-Vto setLD_LIBRARY_PATH.Testingpy.test-s.dummy queueTo test ourmap()we provide a dummyqsub,qstat, andqdel.
These are individualpython-scripts which all act on a common state-file intests/resources/dummy_queue_state.jsonin order to fake the sun-grid-engine’s queue.dummy_qsub.pyonly appends queue-jobs to the list of pending jobs in the state-file.dummy_qdel.pyonly removes queue-jobs from the state-file.dummy_qstat.pydoes move the queue-jobs from the pending to the running list, and does trigger the actual processing of the jobs. Each timedummy_qstat.pyis called it performs a single action on the state-file. So it must be called multiple times to process all jobs. It can intentionally bring jobs into the error-state when this is set in the state-file.Before running the dummy-queue, its state-file must be initialized:fromqueue_map_reduceimportdummy_queuedummy_queue.init_queue_state(path="tests/resources/dummy_queue_state.json")When testing ourmap()you set its argumentsqsub_path,qdel_path, andqstat_pathto point to the dummy-queue.Seetests/test_full_chain_with_dummy_qsub.py.Because of the global state-file, only one instance of dummy_queue must run at a time.