Package: pyncette
A reliable distributed scheduler with pluggable storage backends for Async Python.Free software: MIT licenseInstallationMinimal installation (just SQLite persistence):pip install pyncetteFull installation (all the backends and Prometheus metrics exporter):pip install pyncette[all]You can also install the in-development version with:pip install https://github.com/tibordp/pyncette/archive/master.zipDocumentationhttps://pyncette.readthedocs.ioUsage exampleSimple in-memory scheduler (does not persist state)frompyncetteimportPyncette,Contextapp=Pyncette()@app.task(schedule='* * * * *')asyncdeffoo(context:Context):print('This will run every minute')if__name__=='__main__':app.main()Persistent distributed cron using Redis (coordinates execution with parallel instances and survives restarts)frompyncetteimportPyncette,Contextfrompyncette.redisimportredis_repositoryapp=Pyncette(repository_factory=redis_repository,redis_url='redis://localhost')@app.task(schedule='* * * * * */10')asyncdeffoo(context:Context):print('This will run every 10 seconds')if__name__=='__main__':app.main()See theexamplesdirectory for more examples of usage.Use casesPyncette is designed for reliable (at-least-once or at-most-once) execution of recurring tasks (think cronjobs) whose
lifecycles are managed dynamically, but can work effectively for non-reccuring tasks too.Example use cases:You want to perform a database backup every day at noonYou want a report to be generated daily for your 10M users at the time of their choosingYou want currency conversion rates to be refreshed every 10 secondsYou want to allow your users to schedule non-recurring emails to be sent at an arbitrary time in the futurePyncette might not be a good fit if:You want your tasks to be scheduled to run (ideally) once as soon as possible. It is doable, but you will be better served by a general purpose reliable queue like RabbitMQ or Amazon SQS.You need tasks to execute at sub-second intervals with low jitter. Pyncette coordinates execution on a per task-instance basis and this corrdination can add overhead and jitter.Supported backendsPyncette comes with an implementation for the following backends (used for persistence and coordination) out-of-the-box:SQLite (included)Redis (pip install pyncette[redis])PostgreSQL (pip install pyncette[postgres])MySQL 8.0+ (pip install pyncette[mysql])Amazon DynamoDB (pip install pyncette[dynamodb])Pyncette imposes few requirements on the underlying datastores, so it can be extended to support other databases or
custom storage formats / integrations with existing systems. For best results, the backend needs to provide:Some sort of serialization mechanism, e.g. traditional transactions, atomic stored procedures or compare-and-swapEfficient range queries over a secondary index, which can be eventually consistentDevelopmentTo run integration tests you will need Redis, PostgreSQL, MySQL and Localstack (for DynamoDB) running locally.To run the all tests run:toxAlternatively, there is a Docker Compose environment that will set up all the backends so that integration tests can run seamlessly:docker-compose up -d
docker-compose run --rm shell
toxTo run just the unit tests (excluding integration tests):tox -e py310  # or your Python version of choiceNote, to combine the coverage data from all the tox environments run:Windowsset PYTEST_ADDOPTS=--cov-append
toxOtherPYTEST_ADDOPTS=--cov-append toxChangelog0.10.1 (2023-05-09)Include missing lua files in the built wheel0.10.0 (2023-05-08)Drop support for Python 3.7Add support for Python 3.11Modernize Python package structure and lintersFix a few bugs and type annotations0.8.1 (2021-04-08)Improve performance for calculation of the next execution timeAdd ability for repositories to pass a pagination tokenAddadd_to_context()to inject static data to contextClean up documentation and add additional examples0.8.0 (2021-04-05)Added Amazon DynamoDB backendAdded MySQL backendAdded support for partitioned dynamic tasks0.7.0 (2021-03-31)Added support for automatic and cooperative lease heartbeatingPostgreSQL backend can now skip automatic table creationImproved signal handlingCI: Add Codecov integrationDevenv: Run integration tests in Docker Compose0.6.1 (2020-04-02)Optimize the task querying on Postgres backendFix: ensure that there are no name colissions between concrete instances of different dynamic tasksImprove fairness of polling tasks under high contention.0.6.0 (2020-03-31)Added PostgreSQL backendAdded Sqlite backend and made it the default (replacingInMemoryRepository)Refactored test suite to cover all conformance/integration tests on all backendsRefactored Redis backend, simplifying the Lua scripts and improving exceptional case handling (e.g. tasks disappearing between query and poll)Main loop only sleeps for the rest of remainingpoll_intervalbefore next tick instead of the full amountGeneral bug fixes, documentation changes, clean up0.5.0 (2020-03-27)Fixes bug where a locked dynamic task could be executed again on next tick.poll_task is now reentrant with regards to locking. If the lease passed in matches the lease on the task, it behaves as though it were unlocked.0.4.0 (2020-02-16)Middleware support and optional metrics via PrometheusImproved the graceful shutdown behaviorTask instance and application context are now available in the task contextBreaking change: dynamic task parameters are now accessed viacontext.args[‘name’]instead ofcontext.nameImproved examples, documentation and packaging0.2.0 (2020-01-08)Timezone supportMore efficient poling when Redis backend is used0.1.1 (2020-01-08)First release that actually works.0.0.0 (2019-12-31)First release on PyPI.