Package: saq
SAQSAQ (Simple Async Queue) is a simple and performant job queueing framework built on top of asyncio and redis. It can be used for processing background jobs with workers. For example, you could use SAQ to schedule emails, execute long queries, or do expensive data analysis.DocumentationIt usesredis-py>= 4.2.It is similar toRQand heavily inspired byARQ. Unlike RQ, it is async and thussignificantly fasterif your jobs are async. Even if they are not, SAQ is still considerably faster due to lower overhead.SAQ optionally comes with a simple UI for monitor workers and jobs.Install# minimal install
pip install saq

# web + hiredis
pip install saq[web,hiredis]Usageusage: saq [-h] [--workers WORKERS] [--verbose] [--web]
           [--extra-web-settings EXTRA_WEB_SETTINGS]
           [--port PORT] [--check]
           settings

Start Simple Async Queue Worker

positional arguments:
  settings              Namespaced variable containing
                        worker settings eg: eg
                        module_a.settings

options:
  -h, --help            show this help message and exit
  --workers WORKERS     Number of worker processes
  --verbose, -v         Logging level: 0: ERROR, 1: INFO,
                        2: DEBUG
  --web                 Start web app. By default, this
                        only monitors the current
                        worker's queue. To monitor
                        multiple queues, see '--extra-
                        web-settings'
  --extra-web-settings EXTRA_WEB_SETTINGS, -e EXTRA_WEB_SETTINGS
                        Additional worker settings to
                        monitor in the web app
  --port PORT           Web app port, defaults to 8080
  --check               Perform a health check

environment variables:
  AUTH_USER     basic auth user, defaults to admin
  AUTH_PASSWORD basic auth password, if not specified, no auth will be usedExampleimportasynciofromsaqimportCronJob,Queue# all functions take in context dict and kwargsasyncdeftest(ctx,*,a):awaitasyncio.sleep(0.5)# result should be json serializable# custom serializers and deserializers can be used through Queue(dump=,load=)return{"x":a}asyncdefcron(ctx):print("i am a cron job")asyncdefstartup(ctx):ctx["db"]=awaitcreate_db()asyncdefshutdown(ctx):awaitctx["db"].disconnect()asyncdefbefore_process(ctx):print(ctx["job"],ctx["db"])asyncdefafter_process(ctx):passqueue=Queue.from_url("redis://localhost")settings={"queue":queue,"functions":[test],"concurrency":10,"cron_jobs":[CronJob(cron,cron="* * * * * */5")],# run every 5 seconds"startup":startup,"shutdown":shutdown,"before_process":before_process,"after_process":after_process,}To start the worker, assuming the previous is available in the python pathsaq module.file.settingsNote:module.file.settingscan also be a callable returning the settings dictionary.To enqueue jobs# schedule a job normallyjob=awaitqueue.enqueue("test",a=1)# wait 1 second for the job to completeawaitjob.refresh(1)print(job.results)# run a job and return the resultprint(awaitqueue.apply("test",a=2))# schedule a job in 10 secondsawaitqueue.enqueue("test",a=1,scheduled=time.time()+10)DemoStart the workerpython -m saq examples.simple.settings --webNavigate to theweb uiEnqueue jobspython examples/simple.pyComparison to ARQSAQ is heavily inspired byARQbut has several enhancements.Avoids polling by leveragingBLMOVEorRPOPLPUSHand NOTIFYSAQ has much lower latency than ARQ, with delays of < 5ms. ARQ's default polling frequency is 0.5 secondsSAQ is up to8x fasterthan ARQWeb interface for monitoring queues and workersHeartbeat monitor for abandoned jobsMore robust failure handlingStorage of stack tracesSweeping stuck jobsHandling of cancelled jobs different from failed jobs (machine redeployments)Before and after job hooksEasily run multiple workers to leverage more coresDevelopmentpython -m venv env
source env/bin/activate
pip install -e ".[dev,web]"
docker run -p 6379:6379 redis
./run_checks.sh