Package: matter-task-queue
matter-task-queueThe Task Queue Library is a Python library that provides functionality for managing queues and integrates Celery into FastAPI apps. It enables a seamless integration that unlocks all of Celery's capabilities and allows for easy queue management and task execution.Celery is an open-source, distributed task queue that is widely used in Python-based applications for processing and executing background tasks or long-running processes asynchronously. It was created in 2009 and has since grown into a mature and powerful library that provides a robust and scalable solution for managing task queues.The core idea behind Celery is to divide an application into small, independent tasks that can be executed asynchronously in a distributed environment. Tasks are added to a queue, where they wait to be picked up by a worker process. Once a worker picks up a task, it executes it and updates the status of the task. This allows tasks to be executed independently and in parallel, without blocking the main application thread.Celery provides a number of features that make it a popular choice for managing task queues. It supports a wide range of brokers, including RabbitMQ, Redis, and Amazon SQS, which allows it to work with many different messaging systems. It also provides support for scheduling tasks, task retries, and task prioritization, which makes it easy to build complex workflows.Another key feature of Celery is its ability to distribute tasks across multiple worker nodes. This allows tasks to be processed in parallel, which can improve overall performance and scalability. Celery provides several different strategies for distributing tasks, including round-robin, direct, and fanout.Table of ContentsInstallationLicenseInstallationpip install matter-task-queueMake sure that you have set the following ENV values:ENV=local (otherwise, test-or-development-or-production)DEBUG=falseINSTANCE_NAME=your-webapp-nameCELERY_BROKER_URL=your-broker-urlCELERY_LOG_LEVEL=infoCELERY_LOG_FILE_PATH=/tmp/celery.txtAWS_REGION=eu-central-1SENTRY_DSN=your-sentry-dsn#Foralocalenvironmentonly:AWS_ENDPOINT_URL=http://localhost:9324For a local environment only:
Install and run a docker SQS Queue container:docker run -p 9324:9324 -p 9325:9325 softwaremill/elasticmqUsageTo use the Task Queue Library in your FastAPI app, you can import the TaskQueue class and create an instance of it:# Create apiapp=FastAPI(title="My FastAPI Web Service",root_path=env.PATH_PREFIX,)frommatter_task_queueimportcreate_celeryapp.celery=create_celery(task_module_paths=["your.task.model.paths","",...])The constructor takes two more optional arguments:celery_beat_schedule: The schedule for periodic cron-jobs
create_dead_letter_queue: defaults to TrueContributingMake sure you have all supported python versions installed in your machine:3.103.11Install hatch in your systemhttps://hatch.pypa.io/latest/install/Create the environmenthatch env createDo your changes...Run the testshatch run testThe command above will run the tests against all supported python versions
installed in your machine. For testing in other operating system you may use the
configured CI in github.Bump a new versionIn general, you just need to execute:hatch versionThis command will update the minor version. i.e.:
No breaking changes and new feature has been addedWe are usingsemantic version, if you are doing a bug fix:hatch version fix