Package: dvc-task
dvc-task is a library for queuing, running and managing background jobs
(processes) from standalone Python applications. dvc-task is built onCelery,
but does not require a full AMQP messaging server (or any other “heavy” servers
which are traditionally used as Celery brokers).Featuresdvc_task.procmodule for running and managing background processes in
Celery tasksPreconfigured Celery app intended for use in standalone desktop
applicationsUsesKombufilesystem transport as the message broker, and the standard
filesystem Celery results backendAllows standalone applications to make use of Celery without the use of
additional messaging and results backend serversPreconfigured “temporary” Celery worker which will automatically terminate
itself when the Celery queue is emptyAllows standalone applications to start Celery workers as needed directly
from Python code (as opposed to requiring a “run-forever” daemonized
CLIceleryworker)RequirementsCelery 5.3 or laterKombu 5.3 or laterNote: Windows is not officially supported in Celery, but dvc-task is tested on
Windows (and used in DVC on Windows).InstallationYou can installdvc-taskviapipfromPyPI:$pipinstalldvc-taskUsageProcesses (dvc_task.proc)The process module provides a simple API for managing background processes in
background tasks. Background processes are run in Celery tasks, but process
state is stored separately from Celery, so information about managed processes
can be accessed from outside of the Celery producer or consumer application.After you have configured a Celery application, jobs can be queued (and run) viaProcessManager.run(which returns a signature for theproc.tasks.runCelery task):fromdvc_task.procimportProcessManager@app.taskdefmy_task():manager=ProcessManager(wdir=".")manager.run(["echo","hello world"],name="foo").delay()TheProcessManagerwill create a subdirectory inwdirfor each managed process.$ tree .
.
└── 25mYD6MyLNewXXdMVYCCr3
    ├── 25mYD6MyLNewXXdMVYCCr3.json
    ├── 25mYD6MyLNewXXdMVYCCr3.out
    └── 25mYD6MyLNewXXdMVYCCr3.pid
1 directory, 3 filesAt a minimum, the directory will contain<id>.pidand<id>.jsonfiles.<id>.json: A JSON file describing the process containing the following dictionary keys:pid: Process PIDstdout: Redirected stdout file path for the process (redirected to<id>.outby default)stderr: Redirected stderr file path for the process (stderr is
redirected tostdoutby default)stdin: Redirected stdin file path for the process (interactive
processes are not yet supported, stdin is currently alwaysnull)returncode: Return code for the process (nullif the process
has not exited)<id>.pid: A standard pidfile containing only the process PIDProcessManagerinstances can be created outside of a Celery task to manage
and monitor processes as needed:>>>fromdvc_task.procimportProcessManager>>>manager=ProcessManager(wdir=".")>>>names=[nameforname,_infoinmanager.processes()]['25mYD6MyLNewXXdMVYCCr3']>>>forlineinmanager.follow(names[0]):...print(line)...helloworldCelery Workers (dvc_task.worker)dvc-task includes a pre-configured Celery worker (TemporaryWorker) which
can be started from Python code. TheTemporaryWorkerwill consume Celery
tasks until the queue is empty. Once the queue is empty, the worker will wait
up until a specified timeout for new tasks to be added to the queue. If the
queue remains empty after the timeout expires, the worker will exit.To instantiante a worker with a 60-second timeout, with the Celery worker namemy-worker-1:>>>fromdvc_task.workerimportTemporaryWorker>>>worker=TemporaryWorker(my_app,timeout=60)>>>worker.start("my-worker-1")Note thatworker.startruns the Celery worker within the calling thread.Celery Applications (dvc_task.app)dvc-task includes a pre-configured Celery application (FSApp) which uses
the Kombu filesystem transport as the Celery broker along with the Celery
filesystem results storage backend.FSAppis intended to be used in
standalone Python applications where a traditional Celery producer/consumer
setup (with the appropriate messaging and storage backends) is unavailable.>>>fromdvc_task.appimportFSApp>>>my_app=FSApp(wdir=".")FSAppprovides iterators for accessing Kombu messages which are either
waiting in the queue or have already been processed. This allows the caller
to access Celery task information without using the CeleryinspectAPI
(which is only functional when a Celery worker is actively running).>>>formsginmy_app.iter_processed():...msg<Messageobjectat0x102e7f0d0withdetails{'state':'RECEIVED','content_type':'application/json','delivery_tag':'0244c11a-1bcc-47fc-8587-66909a55fdc6',...}><Messageobjectat0x1027fd4c0withdetails{'state':'RECEIVED','content_type':'application/json','delivery_tag':'491415d1-9527-493a-a5d7-88ed355da77c',...}><Messageobjectat0x102e6f160withdetails{'state':'RECEIVED','content_type':'application/json','delivery_tag':'ea6ab7a4-0398-42ab-9f12-8da1f8e12a8a',...}><Messageobjectat0x102e6f310withdetails{'state':'RECEIVED','content_type':'application/json','delivery_tag':'77c4a335-2102-4bee-9cb8-ef4d8ef9713f',...}>ContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theApache 2.0 license,dvc-taskis free and open source software.IssuesIf you encounter any problems,
pleasefile an issuealong with a detailed description.