Package: django-eb-sqs-worker
django-eb-sqs-workerDjango Background Tasks for Amazon Elastic Beanstalk.Created by Alexey "DataGreed" Strelkov.Overviewdjango-eb-sqs-workerlets you handle background jobs onElastic Beanstalk Worker Environmentsent via SQS and provides methods to send tasks to worker.You can use the same Django codebase for both your Web Tier and Worker Tier environments and send tasks
from Web environment to Worker environment. Amazon fully manages autoscaling for you.Tasks are sent viaAmazon Simple Queue Serviceand are delivered to your worker withElastic Beanstalk's SQS daemon.
Periodic tasks are also supported.Here's the diagram of how tasks move through the system, tasks movement is represented by arrows:InstallationInstall using pip (only python3.x+ is supported):pip install django-eb-sqs-workerAddeb_sqs_workertosettings.INSTALLED_APPS:INSTALLED_APPS=[# ..."eb_sqs_worker",]Addlocalhosttosettings.ALLOWED_HOSTSso SQS Daemon can post tasks from
the queue to your worker:ALLOWED_HOSTS=[# ..."localhost",]Update yoursettings.pyfor both Worker and Web EB environments:# region where your elastic beanstalk environments are deployed, e.g. "us-west-1"AWS_EB_DEFAULT_REGION="your default region"# your aws access key idAWS_ACCESS_KEY_ID="insert your key id here"# your aws access keyAWS_SECRET_ACCESS_KEY="insert your key here"# queue name to use - queues that don't exist will be created automaticallyAWS_EB_DEFAULT_QUEUE_NAME="any_queue_name_to_use"In the settings file for yourWebtier environment add the following setting
(this is important due to possible security problems if you don't set this):# never set to True on Web environment. Use True only on Worker env and local development envAWS_EB_HANDLE_SQS_TASKS=FalseIn the setting files used by yourWorkerenvironments add the following setting:# never set to True on Web environment. Use True only on Worker env and local development envAWS_EB_HANDLE_SQS_TASKS=TrueAdd eb-sqs-worker urls to your project's mainurls.pymodule:# urls.pyurlpatterns=[# your url patterns# ...]fromeb_sqs_worker.urlsimporturlpatternsaseb_sqs_urlpatternsurlpatterns+=eb_sqs_urlpatternsNavigate to your Worker environment in Elastic Beanstalk Web console, then go toConfiguration > Workerand set HTTP path to/sqs/.You should also select the queue to use here corresponding to yourAWS_EB_DEFAULT_QUEUE_NAMEor, if you prefer to use the autogenerated one, you can copy its name and set as yourAWS_EB_DEFAULT_QUEUE_NAME.
If you don't see yourAWS_EB_DEFAULT_QUEUE_NAMEhere, try sending first task to it (see"Queueing tasks"section)
and it will be automatically created for you (you may need to reload the page for it to appear here).Apply changes.UsageSimple wayDefining Background TasksTo define a job create a function decorated bytaskdecorator:fromeb_sqs_worker.decoratorsimporttask@taskdefsome_task(**kwargs):# define your task hereprint(f"The decorated test task is being run with kwargs{kwargs}and will echo them back")returnkwargsMake sure the module with your tasks is imported so they will register correctly.The best practice is to do it as soon as django loads, e.g. in your app'smodels.pyor in correspondingAppConfig.Queueing tasksIf the task was defined using@taskdecorator, you can send it to background queue like this:# sends the task to SQS queue where it will be automatically picked up and executed by workersome_task(foo="bar")You can setsettings.AWS_EB_RUN_TASKS_LOCALLYtoTruein development – this will force all tasks to execute
locally in sync mode without sending them to the queue. This is useful for testing.If you need to execute the function synchronously just one time somewhere in your code without changing this setting,
you can do it like this:# runs the task function synchronously without sending it to the queue and returns resultresult=some_task.execute(foo="bar")Note:don't supply positional arguments to the task, always use keyword arguments.Periodic tasksPeriodic tasks are defined the same way as regular task, but it's better to supply a custom name for them:fromeb_sqs_worker.decoratorsimporttask@task(task_name="some_periodic_task")defperiodic_task():# define your periodic task hereprint(f"Periodic test task is being run ")returnTrueAddcron.yamlto the root of the project:version:1cron:-name:"some_periodic_task"url:"/sqs/"schedule:"023***"Deploy your project to elastic beanstalk and your task will run every day at 23:00.Refer tothe documentationfor more info on periodic tasks.Note: periodic tasks don't support arguments passing#TODO describe(add link tohttps://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-periodictasks), explain configurationAdvanced wayDefining Background Tasks#TODO describeQueueing tasks#TODO describePeriodic tasks#TODO describeInterface reference@task decorator#TODO: add info on argumentssend_task#TODO: add info on argumentsSettingsAWS_EB_HANDLE_SQS_TASKSIf set toTrue, tasks will be accepted and handled on this instance. If set toFalse, the URL for handling
tasks will return 404. Defaults toFalse.Important:set this toTrueonlyon yourWorker environmentAWS_EB_ENABLED_TASKSDictionary of enabled tasks. Routes task names to actual task methods.If you register your tasks using thetaskdecorator, you don't need to worry about this setting at all,
it will be set automatically by the decorator.E.g.:AWS_EB_ENABLED_TASKS={# name used in serialization   # path to actual method that does the job"accounts_confirmation_email":"accounts.tasks.send_confirmation_email","analytics_track_event":"analytics.tasks.track_event"}AWS_EB_DEFAULT_REGIONDefault Elastic Beanstalk Region. Use the one that your app id deployed in.AWS_EB_DEFAULT_QUEUE_NAMEName of the queue used by default. If the queue with specified name does not exist, it will be created
automatically when the first task is queued.AWS_ACCESS_KEY_IDAmazon Access Key Id, refer tothe docsAWS_SECRET_ACCESS_KEYAmazon Secret Access Key, refer tothe docsAWS_EB_RUN_TASKS_LOCALLYIf set to true, all tasks will be run locally and synchronnously instead of being sent to SQS Queue. Defaults toFalseAWS_EB_ALERT_WHEN_TASK_EXECUTES_LONGER_THAN_SECONDSSet this to the maximum number of seconds the job (not periodic, useAWS_EB_ALERT_WHEN_PERIODIC_TASK_EXECUTES_LONGER_THAN_SECONDSfor periodic jobs) is supposed to run. If the job finishes requires more time to finish
ADMINS will be notified by email.AWS_EB_ALERT_WHEN_PERIODIC_TASK_EXECUTES_LONGER_THAN_SECONDSSame asAWS_EB_ALERT_WHEN_TASK_EXECUTES_LONGER_THAN_SECONDSbut for periodic jobs, since sometimes they need a separate threshold.SecurityAlways setAWS_EB_HANDLE_SpipQS_TASKS=Falseon Web Tier Environment so the tasks could not be spoofed!
Web Tier environments are typically used for hosting publici websites and can be accessed by anoyone on the Internet,
meaning that anyone can send any jobs to your site if you leave this option on on Web environment.Worker environments can only be accessed internally, e.g. via SQS Daemon that POSTs, soAWS_EB_HANDLE_SQS_TASKS=Trueshould be set only on worker environments.UseElastic Beanstalk Environment propertiesto supply different setting files for Web and Worker environments. See also:docs on designating the Django settingsTips#TODOAccessing Web Tier Database from WorkerYou will probably want your worker environment to have access to the same database as your web tier environment.Assuming you have a web tier environment and a worker environment with the same Django apps deployed
(if you don't have a worker environment, yet, you can create it usingeb create -t worker <environment name>)
and the web tier environment has anattached databaseset up via Elastic Beanstalk with database connection settings populated from environmantal variables,
do the following:Open Elastic Beanstalk Web ConsoleNavigate to your Web Tier environment > Configuration > DatabaseCopy database connection settings. Note that the database password will not be shown here. If you don't remember it,
you can connect to the Web environment usingeb sshand getting it usingcat /opt/python/current/envNavigate to your Worker environment >  Configuration > Software > EditAdd environmental variables for DB connection that you've copied (RDS_PORT,RDS_PASSWORD,RDS_USERNAME,RDS_DB_NAME,RDS_HOSTNAME) and hit "Apply"Navigate to your Worker environment >  Configuration > Instances > EditAdd security group corresponding to your Web Tier environment and hit "Apply", confirm changes.Re-deploy the application usingeb deployto make sure that everything works as expected.Delay abstraction#TODOUsing different cron files for different environments#TODOTestingSynchronous modeWhen developing on local machine it might be a good idea to setAWS_EB_RUN_TASKS_LOCALLY=True, so all the tasks
that should normally be sent to queue will be executed locally on the same machine in sync mode. This lets you test
your actual task methods in integration tests.Testing django-eb-sqs-worker itselfClone the repository.git clone https://github.com/DataGreed/django-eb-sqs-worker.gitInstall requirements (use python virtual environment)cd django-eb-sqs-worker
pip install -r requirements.txtRun testssh test.shContributingIf you would like to contribute, please make a Pull Request with the description of changes and add tests to cover
these changes.Feel free to open issues if you have any problems or questions with this package.TODOstake advantage of the newenvironment link featureadd pickle serializationfinish readmeparse GET-parameters for periodic tasks?Search tagsDjango Elastic Beanstalk Worker Web Tier Asynchronous celery async django-q Jobs Background Tasks SQS