Package: django-celery-token-bucket
Django celery token bucketA dynamictoken bucketimplementation using the database schedulerdjango celery beat.How it worksThe bucket is represented by a celery queue that will not be processed by a worker but just hold our tokens (messages).
Whenever a rate limited task should be run, the decorator tries to consume a message from that queue. If the queue is empty, the task gets retried after the defined timeout.A periodic task will then refill the bucket with tokens whenever they should be available again.Define a token bucketBuckets are defined in the Django config.Following example allows one thousand tokens per hour to throttle access to a rate limited third party API.Add tosettings.pyof your project.fromtypingimportDictfromceleryimportschedulesfromdjango_celery_token_bucketimportTokenBucketINSTALLED_APPS=[...,'django_celery_token_bucket']CELERY_TOKEN_BUCKETS:Dict[str,TokenBucket]={"my_api_client":TokenBucket(name="my_api_client",schedule=schedules.crontab(minute=0),# once every houramount=1000,maximum=1000,)}nameThe name must only consist of letters, numbers and the underscore character as it's used in the name of the celery
queue. It should also be the same as the key in the CELERY_TOKEN_BUCKETS dictionary.scheduleAcelery.schedules.crontabthat defines when the tokens should be refilled.amountThe amount of tokens to add whenever the scheduled refill is run.maximumThe maximum amount of tokens our bucket can hold.token_refill_queueOverride this setting if you want token refill tasks for this bucket to be placed on a specific queue.
This field is by default None. If no value is provided, theCELERY_DEFAULT_QUEUEsetting is used orcelerySync period tasks to refill the bucketsA management commandtoken_bucket_register_periodic_tasksis provided that should be run during deployment of your
application to sync the period tasks and make sure that buckets get properly refilled.Use the rate_limit decoratorThe decorator will make sure that the task that gets decorated will not exceed the limit of available tokens.Decorated tasks must always beboundto allow access to the task instance.frommy_app.celeryimportcelery_appfromdjango_celery_token_bucket.decoratorsimportrate_limit@celery_app.task(bind=True)@rate_limit(token_bucket="my_api_client",countdown=300)defmy_tasK(self,*args,**kwargs):returntoken_bucketName of the token bucket to consume from. Must be defined in the settings (see above) or will fail with an Exception.countdownTime to wait in seconds before the next try when no token is available.affect_task_retriesDefaults toFalseBy default a failed token retrieval will not impact the retry count of your task. To change this behavior, setaffect_task_retriestoTrue.@celery_app.task(bind=True,max_retries=3,countdown=60)@rate_limit(token_bucket="my_api_client",countdown=300,affect_task_retries=True)defmy_tasK(self,*args,**kwargs):returnIn this scenario, a failed token retrieval will increase the retry count of the task decorator.
If we cannot get a token on the first try, we will start over again with the 2nd try.Run the tests locallyA docker compose environment is provided to easily run the tests:dockercomposerun--rmapptestMaking a new releasebumpversionis used to manage releases.Add your changes to theCHANGELOG, runbumpversion<major|minor|patch>and push (including tags).