Package: just-jobs
just-jobsA friendly and lightweight wrapper forarq. just-jobs provides a simple interface on top of arq that implements additional functionality like synchronous job types (IO-bound vs. CPU-bound) and signed and secure task serialization.Documentation:https://justjobs.thearchitector.dev.Tested support on Python 3.7, 3.8, 3.9, and 3.10, 3.11.$pdmaddjust-jobs# or$pipinstall--userjust-jobsFeaturesjust-jobs doesn't aim to replace the invocations that arq provides, only wrap some of them to make job creation and execution easier and better. It lets you:Define and run non-async jobs. Passing a non-async@jobfunction to arq will run properly. Non-async jobs can also be defined as either IO-bound or CPU-bound, which changes how the job will be executed to prevent blocking the asyncio event loop.The arqContextparameter now works a lot likeFastAPI'sRequest. It's no longer a required parameter, but if it exists, it will get set. It doesn't have to be namedctxeither, only have the typeContext.Specify a singleRedisSettingswithin yourWorkerSettingsfrom which you can create a pool usingSettings.create_pool().Run jobs either immediately or via normal arq enqueueing.Use non-pickable job arguments and kwargs (supported by thedilllibrary).UsageUsing just-jobs is pretty straight forward:Add@job()to any function to make it a delayable job.If the job is synchronous, specify its job type so just-jobs knows how to optimally run it. If you don't, you'll get an error. This helps encourage thoughtful and intentional job design while ensuring that the event loop is never blocked.@job(job_type=JobType.CPU_BOUND)defcomplex_math(i:int,j:int,k:int)If it's a coroutine function, you don't need to specify a job type (and will get a warning if you do).@job()asyncdefpoll_reddit(subr:str)Invoke a job normally if you want to run it immediately.Invoking a job as a regular function allows you to run a job as if it were one. If you have logic that you only want to execute when enqueued, include a parameter with typeContextand check if it exists at runtime (functions with aContextthat are run immediately will have that argument set toNone).@job()asyncdefcontext_aware(ctx:Context,msg:str):ifctx:# enqueued then run by arqreturnf"hello{msg}"else:# invoked manuallyreturnf"bye{msg}"awaitcontext_aware("world")=="bye world"j=awaitp.enqueue_job("context_aware","world")awaitj.result()=="hello world"Define WorkerSettings using theBaseSettingsmetaclass.The execution logic that@jobprovides requires some stuff. When you defining your WorkerSettings, you must declareBaseSettingsas its metaclass to ensure that stuff exists.classSettings(metaclass=BaseSettings):redis_settings=...UseSettings.create_pool().While you may elect to usearq.connections.create_poolas you would normally, using thecreate_poolfunction provided by yourSettingsclass ensures the pool it creates always matches your worker's Redis and serialization settings (it will be less of a headache). It also lets you take advantage of additional functionality, namely that it can be used as an auto-closing context manager.# manuallypool=awaitSettings.create_pool()awaitpool.close(close_connection_pool=True)# or as an async context managerasyncwithSettings.create_pool()aspool:...Enqueue your job.just-jobs doesn't change the way in which you enqueue your jobs. Just useawait pool.enqueue_job(...).awaitpool.enqueue_job('complex_math',2,1,3)Caveatsarq.func()and@job()are mutually exclusive. If you want to configure a job in the same way, pass the settings you would have passed tofunc()to@job()instead.@job(job_type=JobType.CPU_BOUND,keep_result_forever=True,max_tries=10)deftask(a:int,b:int):returna+bThere isn't support for asynchronous CPU-bound tasks. Currently, job types only configure the execution behavior of synchronous tasks (not coroutines). However, there are some valid cases for CPU-bound tasks that also need to be run in an asyncio context.At the moment, the best way to achieve this will be to create a synchronous CPU-bound task (so it runs in a separate process) that then invokes a coroutine viaasyncio.run. If you intend on running the task in the current context from time to time, just return the coroutine instead and it will get automatically executed in the current event loop.async_async_task(a:int,b:int,c:int):ab=awaitadd(a,b)returnawaitadd(ab,c)@job(job_type=JobType.CPU_BOUND)defwrapper_cpu_bound(ctx:Context,a:int,b:int,c:int):task=_async_task(a,b,c)returnasyncio.run(task)ifctxelsetaskExampleThe complete example is available atdocs/example.pyand should work out of the box. The snippet below is just an excerpt to show the features described above:fromjust_jobsimportBaseSettings,Context,JobType,job@job()asyncdefasync_task(url:str):returnurl@job(job_type=JobType.IO_BOUND)defsync_task(ctx:Context,url:str):# if the context is present, this is being run from the arq listenerifctx:print(url)returnurlclassSettings(metaclass=BaseSettings):functions=[async_task,sync_task]redis_settings=RedisSettings(host="redis")asyncdefmain():# create a Redis pool using the Settings already definedpool=awaitSettings.create_pool()# run the_task right now and return the urlurl=sync_task("https://www.theglassfiles.com")awaitpool.enqueue_job("async_task","https://www.eliasfgabriel.com")awaitpool.enqueue_job("sync_task","https://gianturl.net")awaitpool.close(close_connection_pool=True)LicenseThis software is licensed under the3-Clause BSD License.This package isTreeware. If you use it in production, considerbuying the world a treeto thank me for my work. By contributing to my forest, you’ll be creating employment for local families and restoring wildlife habitats.