Package: beanstalk-dispatch
beanstalk-dispatchbeanstalk-dispatchis a Django application that runs functions that have been
scheduled to run anAWS SQSqueue and executes
them onElastic BeanstalkWorker
machines that are listening to that queue.This application was originally written by@marcuafor@b12io's open source applicationorchestra.The library supports Django 3 to Django 4 across Python versions 3.6 to
3.10. If you would like to see a feature or find a bug, please let me know by
opening anissueorpull request.Getting started in 5 minutesTo install:pip install beanstalk-dispatchcreate an Elastic Beanstalk environment for an application
that has the following two parameters insettings.py:BEANSTALK_DISPATCH_SQS_KEY='your AWS key for accessing SQS'BEANSTALK_DISPATCH_SQS_SECRET='your AWS secret for accessing SQS'Addbeanstalk_dispatchto settings.py'sINSTALLED_APPSINSTALLED_APPS=(# ...other installed applications...'beanstalk_dispatch',)Addurl(r'^beanstalk_dispatch/', include('beanstalk_dispatch.urls')),to
your mainurls.pyAdd/beanstalk_dispatch/dispatcheras the HTTP endpoint or your beanstalk
worker configuration in the AWS console.Add a dispatch table. The dispatcher works by creating an HTTP endpoint
that a local SQS/Beanstalk daemon POSTs requests to. That endpoint
consults aBEANSTALK_DISPATCH_TABLE, which maps function names onto
functions to run. Here's an example:ifos.environ.get('BEANSTALK_WORKER')=='True':BEANSTALK_DISPATCH_TABLE={'a_function_to_dispatch':('some_package.beanstalk_tasks.''the_name_of_the_function_in_the_module')}The first line is a check we have that ensures this type of machine should
be a beanstalk worker. We set aBEANSTALK_WORKERenvironment variable to'True'in the environment's configuration only on our worker machines.
This avoids other environments (e.g., our web servers) from serving as open
proxies for running arbitrary code.The second line is the dispatch table. It maps a path to the function to be
executed.Scheduling a function to runThebeanstalk_dispatch.client.schedule_functionschedules a function to run
on a given SQS queue. The function name you pass it must be a key in theBEANSTALK_DISPATCH_TABLE, and thequeue_nameyou pass it must be a queue
for which a beanstalk worker is configured.frombeanstalk_dispatch.clientimportschedule_functionschedule_function('a-queue','a_function_to_dispatch','1','2',kwarg1=1,kwarg2=2)SafeTasksBy default, every function run bybeanstalk_dispatchis wrapped in aSafeTaskclass that sets a@timeoutdecorator on the function and catches
any exceptions for logging. If you would like to customize the behavior of theSafeTask, create a subclass and reference this object inBEANSTALK_DISPATCH_TABLE.The following parameters/functions are configurable on aSafeTasktimeout_timedelta: maximum number of seconds task can run, defaults to2minutes.verbose: boolean specifying if failures are logged, defaults toFalse.run: abstract method to fill in with task work.on_error: a function that runs if the task fails for any reason.on_success: a function that runs after the task completes successfully.on_completion: a function that runs after each task (afteron_errororon_success).For example:# beanstalk_tasks.pyfromdatetimeimporttimedeltafrombeanstalk_dispatch.clientimportschedule_functionfrombeanstalk_dispatch.safe_taskimportSafeTaskclassMySafeTask(SafeTask):timeout_timedelta=timedelta(seconds=1000)verbose=Truedefrun(self,*args,**kwargs):# Run the taskprint('Running task')defon_error(self,e,*args,**kwargs):print('There was an error{}'.format(e))defon_success(self,*args,**kwargs):print('Success!')defon_completion(self,*args,**kwargs):print('Task completed')schedule_function('a-queue','mysafetask','1','2',kwarg1=1,kwarg2=2)# settings.py
  if os.environ.get('BEANSTALK_WORKER') == 'True':
    BEANSTALK_DISPATCH_TABLE = {
        'mysafetask': 'beanstalk_tasks.MySafeTask'
    }