Package: celery-sqlalchemy-kit
celery-sqlalchemy-kitAboutThis kit enables you to store periodic celery tasks in an SQLAlchemy compatible database.
The schedules can be set ascrontabsortime-intervals.
Scheduled tasks in the database can beset active or inactiveto control whether they should run.This kit also allows your celery workers torun asynchronous tasks.This package is under active development and used in production at cap-on.NOTE: This package was developed and tested with a PostgreSQL backend.
Theoretically, any other SQLAlchemy compatible database, that supports JSON type, can be used.
But this has not been tested yet.Getting StartedRequirementspython >= 3.10celery >= 5.2.7sqlalchemy >= 1.4.46  - 2.0 should work as wellpsycopg2 >= 2.9.3 / mysql-connector / other connector depending on databaseredis >= 4.5.1 / other broker/backend for celeryEarlier versions should work as well, they just have not been tested yet.InstallationYou can install this package from PyPi:pipinstallcelery-sqlalchemy-kitSQL Table 'routines'Using this package will create a table that contains your scheduled celery tasks.
The structure of table 'routines' is as follows:ColumnTypeNullableiduuidnot nullnamecharacter varying(50)not nulltaskcharacter varying(50)not nullschedulejsonnot nulllast_run_attimestamp without time zonetotal_run_countintegeractivebooleannot nullkwargsjsonoptionsjsonUsage & ConfigurationTo demonstrate how to use the features of this package, there are examples in the 'example'-directory.
These are being explained in the following.1. ConfigurationFirst you need to instantiate celery, as you would normally do.
Have a look at thecelery documentation.If you define your tasks in another file, include the path to their file viainclude.celery=Celery("celery",include=["celery-sqlalchemy-kit.example.custom_tasks"],backend=your_result_backend,broker=your_broker_url)To use this package you have to configure some variables of your celery instance like so:celery.conf.update({"scheduler_db_uri":scheduler_db_uri,"scheduler_max_interval":scheduler_max_interval,"scheduler_sync_every":scheduler_sync_every,"celery_max_retry":celery_max_retry,"celery_retry_delay":celery_retry_delay,"create_table":True,},)variableexplanationdefaultscheduler_db_uridb uri used by scheduler (must be synchronous)"psycopg2:///schedule.db"scheduler_max_intervalmaximum time to sleep between re-checking the schedule300 (seconds)scheduler_sync_everyHow often to sync the schedule3 * 60 (seconds)celery_max_retryHow often to retry a task when it fails3celery_retry_delayHow long to wait before next retry of failed task is started300 (seconds)create_tableIf setTrue, table 'routines' for scheduled tasks is created automatically with sqlalchemy. If you wish to use alembic, set toFalseTrueMake sure to use the correctscheduler_db_uriof your project allowing theRoutineSchedulerto create a table namedroutinesand save your scheduled tasks in it.
These variables are also available as environment variables in upper case (except forcreate_table).2.1. Create scheduled tasksTo create tasks that run after your desired schedule, you have to inherit from classSyncTasklike so:classCeleryTestTask(SyncTask):name="celery test"schedule=15defrun(self,*args,**kwargs):# do stuffThe task that you want to be executed, has to be defined as therunmethod from your task class. Make sure to define a name for your task and set the variableschedule. For the schedule you have to options:schedule typeexplanationexamplesyntax in dbtime intervalrun your task every ... secondsschedule = 15{"timedelta": 15}crontabdefine schedule as crontab, by creating a dictschedule = {"minute": 0, "hour": 9, "day_of_month": 15}{"minute": 0, "hour": 9, "day_of_month": 15}Now when you start your program, the scheduled tasks are added to your database inroutinestable and executed by a celery worker within the schedule you defined.NOTE: In your database the schedules of tasks are saved as type JSON.
Make sure to keep the correct syntax.2.2. Create asynchronous scheduled tasksThis works the same way as with synchronous tasks, except that your custom tasks has to inherit from classAsnycTaskand the task logic has to be implemented within the async methodexecute:classCeleryTestTask(AsyncTask):name="celery test"schedule=15asyncdefexecute(self,*args,**kwargs):# do stuff3. Change schedule / (in-) activate tasksIf you wish to change the schedule of a task, just update the corresponding db entry.
The next time theRoutineSchedulersynchronizes, it will acknowledge the new schedule.
Same thing with activating or inactivating tasks.
To activate a task, set columnactivein your db tot(True).
To inactivate a task, set columnactivein your db tof(False).A task that is inactive will not be executed as long as you change it to active again.4. Source of truth for scheduleNew scheduled task in code, that is not in db: new db entry is created automaticallyScheduled task in code as well as db: schedule in db is used to run taskTasks that are deleted in code will be removed from db after redeployInactive tasks in db will not be executed5. Run celery worker and beatTo start celery worker:$celery-Acelery_instanceworker--loglevel=INFOTo start celery beat:$celery-Acelery_instancebeat--loglevel=INFO--scheduler=scheduler.RoutineSchedulercelery_instanceis the file containing your celery instance.
Use the correct path of your project.