Package: django-eb-sqs
Django EB SQS - Background Tasks for Amazon SQSdjango-eb-sqs is a simple task manager for AWS SQS. It uses SQS and theboto3library.InstallationInstall the module withpip install git+git://github.com/cuda-networks/django-eb-sqs.gitor add it to yourrequirements.txt.Don't forget to add django-eb-sqs app to your DjangoINSTALLED_APPSsettings:INSTALLED_APPS=(...,'eb_sqs',)UsageCreating TasksAdding a task to a queue is simple.fromeb_sqs.decoratorsimporttask@task(queue_name='test')defecho(message):print(message)echo.delay(message='Hello World!')NOTE:This assumes that you have your AWS keys in the appropriate environment variables, or are using IAM roles. Consult theboto3documentationfor further info.If you don't pass a queue name, theEB_SQS_DEFAULT_QUEUEsetting is used. If not set, the queue name iseb-sqs-default.Additionally the task decorator supportsmax_retries(default0) anduse_pickle(defaultFalse) attributes for advanced control task execution.You can also delay the execution of a task by specifying the delay time in seconds.echo.delay(message='Hello World!',delay=60)During development it is sometimes useful to execute a task immediately without using SQS. This is possible with theexecute_inlineargument.echo.delay(message='Hello World!',execute_inline=True)NOTE:delayis not applied whenexecute_inlineis set toTrue.Failed tasks can be retried by using theretrymethod. See the following example:fromeb_sqs.decoratorsimporttask@task(queue_name='test',max_retries=5)defupload_file(message):print('# of retries:{}'.format(upload_file.retry_num))try:# upload ...exceptConnectionException:upload_file.retry()The retry call supports thedelayandexecute_inlinearguments in order to delay the retry or execute it inline. If the retry shall not be counted for the max retry limit setcount_retriesto false. Use 'retry_num' to get the number of retries for the current task.NOTE:retry()throws aMaxRetriesReachedExceptionexception if the maximum number of retries is reached.Executing TasksIn order to execute tasks, use the Django commandprocess_queue.
This command can work with one or more queues, reading from the queues infinitely and executing tasks as they come-in.pythonmanage.pyprocess_queue--queues<comma-delimitedlistofqueuenames>You can either use full queue names, or queue prefix usingprefix:*my_example_prefix*notation.Examples:pythonmanage.pyprocess_queue--queuesqueue1,queue2# process queue1 and queue2pythonmanage.pyprocess_queue--queuesqueue1,prefix:pr1-,queue2# process queue1, queue2 and any queue whose name starts with 'pr1-'Use the signalsMESSAGES_RECEIVED,MESSAGES_PROCESSED,MESSAGES_DELETEDof theWorkerServiceto get informed about the current SQS batch being processed by the management command.Auto TasksThis is a helper tool for the case you wish to define one of your class method as a task, and make it seamless to all callers.
This makes the code much simpler, and allows using classes to invoke your method directly without considering whether it's invoked async or not.This is how you would define your class:fromeb_sqs.auto_tasks.serviceimportAutoTaskServiceclassMyService:def__init__(self,p1=default1,...,pN=defaultN,auto_task_service=None):self._auto_task_service=auto_task_serviceorAutoTaskService()self._auto_task_service.register_task(self.my_task_method)defmy_task_method(self,*args,**kwargs):...Notice the following:Your class needs to have defaults for all parameters in the c'torThe c'tor must have a parameter namedauto_task_serviceThe method shouldn't have any return value (as it's invoked async)In case you want your method to retry certain cases, you need to raiseRetryableTaskException.
You can provide on optionaldelaytime for the retry, setcount_retries=Falsein case you don't want to limit retries, or usemax_retries_functo specify a function which will be invoked when the defined maximum number of retries is exhausted.SettingsThe following settings can be used to fine tune django-eb-sqs. Copy them into your Djangosettings.pyfile.EB_AWS_REGION (us-east-1): The AWS region to use when working with SQS.EB_SQS_MAX_NUMBER_OF_MESSAGES (10): The maximum number of messages to read in a single call from SQS (<= 10).EB_SQS_WAIT_TIME_S (2): The time to wait (seconds) when receiving messages from SQS.NO_QUEUES_WAIT_TIME_S (5): The time a workers waits if there are no SQS queues available to process.EB_SQS_AUTO_ADD_QUEUE (False): If queues should be added automatically to AWS if they don't exist.EB_SQS_QUEUE_MESSAGE_RETENTION (1209600): The value (in seconds) to be passed to MessageRetentionPeriod parameter, when creating a queue (only relevant in case EB_SQS_AUTO_ADD_QUEUE is set to True).EB_SQS_QUEUE_VISIBILITY_TIMEOUT (300): The value (in seconds) to be passed to VisibilityTimeout parameter, when creating a queue (only relevant in case EB_SQS_AUTO_ADD_QUEUE is set to True).EB_SQS_DEAD_LETTER_MODE (False): Enable if this worker is handling the SQS dead letter queue. Tasks won't be executed but group callback is.EB_SQS_DEFAULT_DELAY (0): Default task delay time in seconds.EB_SQS_DEFAULT_MAX_RETRIES (0): Default retry limit for all tasks.EB_SQS_DEFAULT_COUNT_RETRIES (True): Count retry calls. Needed if max retries check shall be executed.EB_SQS_DEFAULT_QUEUE (eb-sqs-default): Default queue name if none is specified when creating a task.EB_SQS_EXECUTE_INLINE (False): Execute tasks immediately without using SQS. Useful during development. Global settingTruewill override setting it on a task level.EB_SQS_FORCE_SERIALIZATION (False): Forces serialization of tasks when executedinline. This setting is helpful during development to see if all arguments are serialized and deserialized properly.EB_SQS_QUEUE_PREFIX (``): Prefix to use for the queues. The prefix is added to the queue name.EB_SQS_USE_PICKLE (False): Enable to usepickleto serialize task parameters. Usesjsonas default.EB_SQS_AWS_MAX_RETRIES (30): Default retry limit on a boto3 call to AWS SQS.EB_SQS_REFRESH_PREFIX_QUEUES_S (10): Minimal number of seconds to wait between refreshing queue list, in case prefix is usedDevelopmentMake sure to install the development dependencies fromdevelopment.txt.TestsThe build in tests can be executed with the Django test runner.python-mdjangotest--settings=eb_sqs.test_settings