Package: tenant-schemas-celery
tenant-schemas-celeryCelery application implementation that allows celery tasks to cooperate with
multi-tenancy provided bydjango-tenant-schemasanddjango-tenantspackages.This project might not seem frequently updated, but it just has all the functionality needed. Issues and questions are answered quickly.Installation$pipinstalltenant-schemas-celery$pipinstalldjango-tenantsUsageDefine a celery app using givenCeleryAppclass.importosos.environ.setdefault('DJANGO_SETTINGS_MODULE','app.settings')fromdjango.confimportsettingsfromtenant_schemas_celery.appimportCeleryAppasTenantAwareCeleryAppapp=TenantAwareCeleryApp()app.config_from_object('django.conf:settings')app.autodiscover_tasks(lambda:settings.INSTALLED_APPS)This assumes a fresh Celery 5.2.0 application. For previous versions, the key is to create a newCeleryAppinstance that will be used to access task decorator from.Replace your@taskdecorator with@app.taskfromceleryimportshared_taskfromdjango.dbimportconnectionfrommyproject.celeryimportapp@app.taskdefmy_task():print(connection.schema_name)@shared_task(base=TenantTask,bind=True)defmy_shared_task():print("foo")Run celery worker (myproject.celeryis where you've defined theappvariable)$celeryworker-Amyproject.celeryPost registered task. The schema name will get automatically added to the task's arguments.frommyproject.tasksimportmy_taskmy_task.delay()TheTenantTaskclass transparently inserts current connection's schema into
the task's kwargs. The schema name is then popped from the task's kwargs intask_prerunsignal handler, and the connection's schema is changed
accordingly.Multiple databases supportNew in2.0.0.Inside your celery tasks you might be working with multiple databases. You might want to change the schema for
all of the connections, or just a subset of them.You can now use theCELERY_TASK_TENANT_CACHE_SECONDSdjango setting, orTASK_TENANT_CACHE_SECONDScelery setting, or
thetenant_databasesattribute of theTenantTaskto a list of database names (the key in thesettings.DATABASESdictionary).If not set, the settings defaults to["default"].Tenant objects cacheNew in0.3.0.Every time a celery task is executed, the tenant object of theconnectionobject is being refetched.
For some use cases, this can introduce significant performance hit.In such scenarios, you can passtenant_cache_secondsargument to the@app.task()decorator. This will
cause the tenant objects to be cached for given period of time.0turns this off. You can also enable cache globally
by setting celery'sTASK_TENANT_CACHE_SECONDS(app-specific, usually it'sCELERY_TASK_TENANT_CACHE_SECONDS).@app.task(tenant_cache_seconds=30)defsome_task():...Celery beat integrationIn order to run celery beat tasks in a multi-tenant environment, you've got two options:Use a dispatching task that will send a task for each tenantUse a custom scheduleri.e: Let's say that you would like to run areset_remaining_jobstasks periodically, for every tenant that you have.Dispatcher task patternYou can schedule one dispatcher task that will iterate over all schemas and send that task within the schema's context:fromdjango_tenants.utilsimportget_tenant_model,tenant_contextfromdjango_tenant_schemas.utilsimportget_tenant_model,tenant_context@app.taskdefreset_remaining_jobs_in_all_schemas():fortenantinget_tenant_model().objects.exclude(schema_name='public'):withtenant_context(tenant):reset_remaining_jobs_in_schema.delay()@app.taskdefreset_remaining_jobs_in_schema():<dosomelogic>Thereset_remaining_jobs_in_all_schemastask (called the dispatch task) should be registered in your celery beat schedule. Thereset_remaining_jobs_in_schematask should be called from the dispatch task.That way you have full control over which schemas the task should be scheduled in.Custom schedulerIf you are using the standardSchedulerorPersistentSchedulerclasses provided bycelery, you can transition to using this package'sTenantAwareSchedulerorTenantAwarePersistentSchedulerclasses. You should then specify the scheduler you want to use in the celery beat config or your invocation tobeat. i.e:celery-Aprojbeat--scheduler=tenant_schemas_celery.scheduler.TenantAwareSchedulerCaveatsThere's a chance that celery beat will try to run a task for a newly created tenant before its migrations are ready, which could potentially lead to deadlocks. This is specially true with big projects with a lot of migrations and very frequent tasks (i.e: every minute). In order to mitigate it, one could do the following:Subclass any ofTenantAwareSchedulerorTenantAwarePersistentSchedulerand override theget_querysetmethod to match your definition of "ready" tenants. For example, imagine that you had areadyflag in your tenant model. You could do the following:# tenants_app/scheduler.pyclassMyTenantAwareScheduler(TenantAwareScheduler):@classmethoddefget_queryset(cls):returnsuper().get_queryset().filter(ready=True)Use the new scheduler in your celery beat config or invocation:celery-Aprojbeat--scheduler=tenants_app.scheduler.MyTenantAwareSchedulerTenantAwareSchedulerMixinuses a subclass ofSchedulerEntrythat allows the user to provide specific schemas to run a task on. This might prove useful if you have a task you only want to run in thepublicschema or to a subset of your tenants. In order to use set that, you must configuretenant_schemasin the tasks definition as such:app.conf.beat_schedule={"my-task":{"task":"myapp.tasks.my_task","schedule":schedules.crontab(minute="*/5"),"tenant_schemas":["public"]}}Compatibility changesThe>=2.1series drop support fortenant-schemas. It hasn't been maintainted for
a long time.The2.xseries support Python>=3.7.The1.xseries support Python>=3.6. Python 3.6 reached EOL 2021-12.The0.xseries are the last one to support Python<3.6.