Package: django-task
1django-taskA Django app to run new background tasks from either admin or cron, and inspect task history from adminContents1django-task1.1Quickstart1.2Features1.3Screenshots1.4App settings1.5Verbosity levels1.6Running Tests1.7Threaded vs RQ-based tasks1.8Support Job class1.9Javascript helpers1.9.1tasks_info_api1.9.2task_add_api1.9.3task_run_api1.10Updating the tasks listing dynamically in the frontend1.11Example Project for django-task1.12Credits2History2.12.0.62.22.0.52.32.0.42.42.0.32.52.0.22.62.0.12.72.0.02.81.5.12.91.5.02.101.4.72.111.4.62.121.4.52.131.4.42.141.4.32.151.4.22.161.4.12.171.4.02.181.3.102.19v1.3.92.20v1.3.82.21v1.3.72.22v1.3.62.23v1.3.52.24v1.3.42.25v1.3.32.26v1.3.12.27v1.3.02.28v1.2.52.29v1.2.42.30v1.2.32.31v1.2.22.32v1.2.12.33v1.2.02.34v1.1.32.35v1.1.22.36v1.1.02.37v0.3.82.38v0.3.72.39v0.3.62.40v0.3.52.41v0.3.42.42v0.3.32.43v0.3.22.44v0.3.12.45v0.3.02.46v0.2.02.47v0.1.152.48v0.1.142.49v0.1.132.50v0.1.122.51v0.1.112.52v0.1.102.53v0.1.92.54v0.1.82.55v0.1.72.56v0.1.62.57v0.1.52.58v0.1.42.59v0.1.32.60v0.1.21.1QuickstartInstall Django Task:pipinstalldjango-taskAdd it to your `INSTALLED_APPS`:INSTALLED_APPS=(...'django_rq',# optional (not needed when using TaskThreaded)'django_task',...)Add Django Task’s URL patterns:urlpatterns=[...path('django_task/',include('django_task.urls',namespace='django_task')),...]Configure Redis and RQ in settings.py(not needed when using TaskThreaded):#REDIS_URL = 'redis://localhost:6379/0'redis_host=os.environ.get('REDIS_HOST','localhost')redis_port=6379REDIS_URL='redis://%s:%d/0'%(redis_host,redis_port)CACHES={'default':{'BACKEND':'redis_cache.RedisCache','LOCATION':REDIS_URL},}## RQ config#RQ_PREFIX="myproject_"QUEUE_DEFAULT=RQ_PREFIX+'default'QUEUE_HIGH=RQ_PREFIX+'high'QUEUE_LOW=RQ_PREFIX+'low'RQ_QUEUES={QUEUE_DEFAULT:{'URL':REDIS_URL,#'PASSWORD': 'some-password','DEFAULT_TIMEOUT':360,},QUEUE_HIGH:{'URL':REDIS_URL,'DEFAULT_TIMEOUT':500,},QUEUE_LOW:{'URL':REDIS_URL,#'ASYNC': False,},}Note: if you plan to install many instances of the project on the same server,
for each instance use a specific value forRQ_PREFIX; for example:INSTANCE_PREFIX="myproject_"try:fromproject.settings.instance_prefiximport*exceptExceptionase:passRQ_PREFIX=INSTANCE_PREFIXQUEUE_DEFAULT=RQ_PREFIX+'_default'QUEUE_LOW=RQ_PREFIX+'_low'QUEUE_HIGH=RQ_PREFIX+'_high'...Customize django-task specific settings (optional):RQ_SHOW_ADMIN_LINK=FalseDJANGOTASK_LOG_ROOT=os.path.abspath(os.path.join(BASE_DIR,'..','protected','tasklog'))DJANGOTASK_ALWAYS_EAGER=FalseDJANGOTASK_JOB_TRACE_ENABLED=FalseDJANGOTASK_REJECT_IF_NO_WORKER_ACTIVE_FOR_QUEUE=TrueOptionally, revoke pending tasks at startapp;filemain/apps.py:classMainConfig(AppConfig):...defready(self):...try:fromdjango_task.utilsimportrevoke_pending_tasksrevoke_pending_tasks()exceptExceptionase:print(e)1.2FeaturesPurposescreate async tasks either programmatically or from adminmonitor async tasks from adminlog all tasks in the database for later inspectionoptionally save task-specific logs in a TextField and/or in a FileFieldDetailseach specific task is described by a Model derived from either models.TaskRQ or models.TaskThreaded, which
is responsible for:selecting the name for the consumer queue among available queues (TaskRQ only)collecting and saving all parameters required by the associated jobrunning the specific job asyncronouslya new job can be run either:creating a Task from the Django admincreating a Task from code, then calling Task.run()job execution workflow:job execution is triggered by task.run(is_async)job will receive the task.id, and retrieve paramerts from iton start, job will update task status to ‘STARTED’ and save job.id for referenceduring execution, the job can update the progress indicatoron completion, task status is finally updated to either ‘SUCCESS’ or ‘FAILURE’See example.jobs.count_beans for an example1.3Screenshots1.4App settingsDJANGOTASK_LOG_ROOTPath for log files.Default: NoneExample: os.path.abspath(os.path.join(BASE_DIR, ‘..’, ‘protected’, ‘tasklog’))DJANGOTASK_ALWAYS_EAGERWhen True, all task are execute syncronously (useful for debugging and unit testing).Default: FalseDJANGOTASK_JOB_TRACE_ENABLEDEnables low level tracing in Job.run() - for debugging challenging race conditionsDefault: FalseDJANGOTASK_REJECT_IF_NO_WORKER_ACTIVE_FOR_QUEUERejects task if not active worker is available for the specific task queue
when task.run() is calledDefault: FalseREDIS_URLRedis server to connect toDefault: ‘redis://localhost:6379/0’1.5Verbosity levelsTheverbosity levelcontrols the logging level as follows:verbositylog level0no log1logging.WARNING2logging.INFO3logging.DEBUGand can be set by the derived class:classMyTask(TaskRQ):...DEFAULT_VERBOSITY=2...or you can set it on a “per task” basis by adding to the model
atask_verbosityfield as follows:task_verbosity=models.PositiveIntegerField(null=False,blank=False,default=2,choices=((0,'0'),(1,'1'),(2,'2'),(3,'3')),)In case, when creating a management command you might want to preserve the default
value provided by the class, or override it if and only if the verbosity option
has been set by the user:importsysdefhandle(self,*args,**options):if'-v'insys.argvor'--verbosity'insys.argv:options['task_verbosity']=options['verbosity']self.run_task(MyTask,**options)1.6Running TestsDoes the code actually work?Running the unit tests from your project:python manage.py test -v 2 django_task --settings=django_task.tests.settingsRunning the unit tests from your local fork:source <YOURVIRTUALENV>/bin/activate
(myenv) $ pip install tox
(myenv) $ toxor:python ./runtests.pyor:coverage run --source='.' runtests.py
coverage report1.7Threaded vs RQ-based tasksThe original implementation is based on django-rq and RQ (a Redis based Python queuing library).On some occasions, using a background queue may be overkill or even inappropriate:
if you need to run many short I/O-bound background tasks concurrently, the serialization
provided by the queue, while limiting the usage of resources, would cause eccessive delay.Starting from version 2.0.0, in those cases you can use TaskThreaded instead of TaskRQ;
this way, each background task will run in it’s own thread.MIGRATING FROM django-task 1.5.1 to 2.0.0derive your queue-based tasks from TaskRQ instead of Taskor use TaskThreadedget_jobclass() overridable replaces get_jobfunc()1.8Support Job classStarting from version 0.3.0, some conveniences have been added:The @job decorator for job functions is no more required, as Task.run() now
uses queue.enqueue() instead of jobfunc.delay(), and retrieves the queue
name directly from the Task itselfeach Task can set it’s own TASK_TIMEOUT value (expressed in seconds),
that when provided overrides the default queue timeouta new Job class has been provided to share suggested common logic before and
after jobfunc execution; you can either overriderun()to implement a custom logic,
or (in most cases) just supply your ownexecute()method, and optionally
overrideon_complete()to execute cleanup actions after job completion;example:classCountBeansJob(Job):@staticmethoddefexecute(job,task):num_beans=task.num_beansforiinrange(0,num_beans):time.sleep(0.01)task.set_progress((i+1)*100/num_beans,step=10)@staticmethoddefon_complete(job,task):print('task "%s" completed with:%s'%(str(task.id),task.status))# An more realistic example from a real project ...# if task.status != 'SUCCESS' or task.error_counter > 0:#    task.alarm = BaseTask.ALARM_STATUS_ALARMED#    task.save(update_fields=['alarm', ])ExecuteRun consumer:pythonmanage.pyrunserverRun worker(s):pythonmanage.pyrqworkerlowhighdefaultpythonmanage.pyrqworkerlowhighdefault...Sample Taskfromdjango.dbimportmodelsfromdjango.confimportsettingsfromdjango_task.modelsimportTaskRQclassSendEmailTask(TaskRQ):sender=models.CharField(max_length=256,null=False,blank=False)recipients=models.TextField(null=False,blank=False,help_text='put addresses in separate rows')subject=models.CharField(max_length=256,null=False,blank=False)message=models.TextField(null=False,blank=True)TASK_QUEUE=settings.QUEUE_LOWTASK_TIMEOUT=60LOG_TO_FIELD=TrueLOG_TO_FILE=FalseDEFAULT_VERBOSITY=2@staticmethoddefget_jobclass():from.jobsimportSendEmailJobreturnSendEmailJobWhen usingLOG_TO_FILE = True, you might want to add a cleanup handler to
remove the log file when the corresponding record is deleted:import os
from django.dispatch import receiver

@receiver(models.signals.post_delete, sender=ImportaCantieriTask)
def on_sendemailtask_delete_cleanup(sender, instance, **kwargs):
    """
    Autodelete logfile on Task delete
    """
    logfile = instance._logfile()
    if os.path.isfile(logfile):
        os.remove(logfile)Sample Jobimportredisimportloggingimporttracebackfromdjango.confimportsettingsfrom.modelsimportSendEmailTaskfromdjango_task.jobimportJobclassSendEmailJob(Job):@staticmethoddefexecute(job,task):recipient_list=task.recipients.split()sender=task.sender.strip()subject=task.subject.strip()message=task.messagefromdjango.core.mailimportsend_mailsend_mail(subject,message,sender,recipient_list)Sample management commandfromdjango_task.task_commandimportTaskCommandfromdjango.contrib.authimportget_user_modelclassCommand(TaskCommand):defadd_arguments(self,parser):super(Command,self).add_arguments(parser)parser.add_argument('sender')parser.add_argument('subject')parser.add_argument('message')parser.add_argument('-r','--recipients',nargs='*')parser.add_argument('-u','--user',type=str,help="Specify username for 'created_by' task field")defhandle(self,*args,**options):fromtasks.modelsimportSendEmailTask# transform the list of recipents into text# (one line for each recipient)options['recipients']='\n'.join(options['recipients'])ifoptions['recipients']isnotNoneelse''# format multiline messageoptions['message']=options['message'].replace('\\n','\n')if'user'inoptions:created_by=get_user_model().objects.get(username=options['user'])else:created_by=Noneself.run_task(SendEmailTask,created_by=created_by,**options)Deferred Task retrieval to avoid job vs. Task race conditionAn helper Task.get_task_from_id() classmethod is supplied to retrieve Task object
from task_id safely.Task queues create a new type of race condition. Why ?
Because message queues are fast !
How fast ?
Faster than databases.See:https://speakerdeck.com/siloraptor/django-tasty-salad-dos-and-donts-using-celeryA similar generic helper is available for Job-derived needs:django_task.utils.get_model_from_id(model_cls, id, timeout=1000, retry_count=10)Howto schedule jobs with cronCall management command ‘count_beans’, which in turn executes the required job.For example:SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 * * * *  {{username}}    timeout 55m {{django.pythonpath}}/python {{django.website_home}}/manage.py count_beans 1000 >> {{django.logto}}/cron.log 2>&1A base class TaskCommand has been provided to simplify the creation of any specific
task-related management commad;a derived management command is only responsible for:defining suitable command-line parametersselecting the specific Task class and job functionfor example:fromdjango_task.task_commandimportTaskCommandclassCommand(TaskCommand):defadd_arguments(self,parser):super(Command,self).add_arguments(parser)parser.add_argument('num_beans',type=int)defhandle(self,*args,**options):fromtasks.modelsimportCountBeansTaskself.run_task(CountBeansTask,**options)1.9Javascript helpersA few utility views have been supplied for interacting with tasks from javascript.1.9.1tasks_info_apiRetrieve informations about a list of existing tasksSample usage:vartasks=[{id:'c50bf040-a886-4aed-bf41-4ae794db0941',model:'tasks.devicetesttask'},{id:'e567c651-c8d5-4dc7-9cbf-860988f55022',model:'tasks.devicetesttask'}];$.ajax({url:'/django_task/info/',data:JSON.stringify(tasks),cache:false,type:'post',dataType:'json',headers:{'X-CSRFToken':getCookie('csrftoken')}}).done(function(data){console.log('data: %o',data);});Result:[
  {
    "id": "c50bf040-a886-4aed-bf41-4ae794db0941",
    "created_on": "2018-10-11T17:45:14.399491+00:00",
    "created_on_display": "10/11/2018 19:45:14",
    "created_by": "4f943f0b-f5a3-4fd8-bb2e-451d2be107e2",
    "started_on": null,
    "started_on_display": "",
    "completed_on": null,
    "completed_on_display": "",
    "job_id": "",
    "status": "PENDING",
    "status_display": "<div class=\"task_status\" data-task-model=\"tasks.devicetesttask\" data-task-id=\"c50bf040-a886-4aed-bf41-4ae794db0941\" data-task-status=\"PENDING\" data-task-complete=\"0\">PENDING</div>",
    "log_link_display": "",
    "failure_reason": "",
    "progress": null,
    "progress_display": "-",
    "completed": false,
    "duration": null,
    "duration_display": "",
    "extra_fields": {
    }
  },
  ...
]1.9.2task_add_apiCreate and run a new task based on specified parametersExpected parameters:‘task-model’ = “<app_name>.<model_name>”… task parameters …Returns the id of the new task.Sample usage:functionexportAcquisition(object_id){if(confirm('Do you want to export data ?')){varurl='/django_task/add/';vardata=JSON.stringify({'task-model':'tasks.exportdatatask','source':'backend.acquisition','object_id':object_id});$.ajax({type:'POST',url:url,data:data,cache:false,crossDomain:true,dataType:'json',headers:{'X-CSRFToken':getCookie('csrftoken')}}).done(function(data){console.log('data: %o',data);alert('New task created: "'+data.task_id+'"');}).fail(function(jqXHR,textStatus,errorThrown){console.log('ERROR: '+jqXHR.responseText);alert(errorThrown);});}return;}1.9.3task_run_apiSchedule execution of specified task.Returns job.id or throws error (400).Parameters:app_labelmodel_namepkis_async (0 or 1, default=1)Sample usage:vartask_id='c50bf040-a886-4aed-bf41-4ae794db0941';$.ajax({url:sprintf('/django_task/tasks/devicetesttask/%s/run/',task_id),cache:false,type:'get'}).done(function(data){console.log('data: %o',data);}).fail(function(jqXHR,textStatus,errorThrown){display_server_error(jqXHR.responseText);});1.10Updating the tasks listing dynamically in the frontendThe list of Tasks in the admin changelist_view is automatically updated to refresh
the progess and status of each running Task.You can obtain the same result in the frontend by calling theDjangoTask.update_tasks()javascript helper, provided you’re listing the tasks in an HTML table with a similar layout.The simplest way to do it is to use therender_task_column_names_as_table_rowandrender_task_as_table_rowtemplate tags.Example:{% load i18n django_task_tags %}

{% if not export_data_tasks %}<div>{% trans 'No recent jobs available' %}</div>{% else %}<tableid="export_data_tasks"class="table table-striped">{% with excluded='created_by,created_on,job_id,log_text,mode' %}<thead><tr>{{ export_data_tasks.0|render_task_column_names_as_table_row:excluded }}</tr></thead><tbody>{% for task in export_data_tasks %}<tr>{{ task|render_task_as_table_row:excluded }}</tr>{% endfor %}</tbody></table>{% endwith %}
{% endif %}


{% block extrajs %}
    {{ block.super }}<scripttype="text/javascript"src="{% static 'js/django_task.js' %}"></script><script>$(document).ready(function(){DjangoTask.update_tasks(1000,'#export_data_tasks');});</script>{% endblock extrajs %}For each fieldname included in the table rows,render_task_as_table_rowwill
check if a FIELDNAME_display() method is available in the Task model, and in case
will use it for rendering the field value; otherwise, the field value will be simply
converted into a string.If the specific derived Task model defines some additional fields (unknown to the base Task model)
which need to be updated regularly byDjangoTask.update_tasks(), include them as “extra_fields”
as follows:defas_dict(self):data=super(ExportDataTask,self).as_dict()data['extra_fields']={'result_display':mark_safe(self.result_display())}returndata1.11Example Project for django-taskAs example project is provided as a convenience feature to allow potential users
to try the app straight from the app repo without having to create a django project.Please follow the instructions detailed in fileexample/README.rst.1.12CreditsReferences:A simple app that provides django integration for RQ (Redis Queue)Asynchronous tasks in django with django-rqdjango-rq redux: advanced techniques and toolsBenchmark: Shared vs. Dedicated Redis InstancesDjango tasty salad - DOs and DON’Ts using Celery by Roberto RosarioCan Django do multi-thread works?2History2.12.0.6fix duration_display()2.22.0.5TaskCommand.run_task() now returns the ID of the created Task2.32.0.4TaskCommand.run_task() now returns the created Task2.42.0.3Prepare for Django 4.02.52.0.2TaskCommand now uses “-v” / “–verbosity” command line options to set task_verbosity2.62.0.1optional “per task” verbosity levelPOSSIBLE INCOMPATIBLE CHANGE: verbosity levels has been shifted (set as documented in the README)2.72.0.0Split Task model into TaskBase + TaskRQImplement TaskThreaded (experimental)Drop Django1 and Python2.7cleanup2.81.5.1Moved required imports inside Job.run() so it can be more easily replicated for any needed customizationSimpler queues settingsRevamped unit testingCleanup2.91.5.0Support for updating the tasks listing dynamically in the frontendExample provided for task_add_api() javascript helperPOSSIBLY INCOMPATIBLE CHANGE: duration and duration_display are now methods rather then propertiesit traslation for UI messages2.101.4.7Added optional “created_by” parameter to TaskCommand utility2.111.4.6replace namespace “django.jQuery” with more generic “jQuery” in js helpersupdate example projectunit tests added to “tasks” app in example project2.121.4.5Quickstart revised in README2.131.4.4Task.get_logger() is now publicly available2.141.4.3restore compatibility with Django 1.11; upgrade rq and django-rq requirements2.151.4.2tasks_info_api() optimized to use a single query2.161.4.1Cleanup: remove redundant REJECTED status2.171.4.0Update requirements (Django >= 2.0, django-rq>=2.0)2.181.3.10Use exceptions.TaskError class when raising specific exceptions2.19v1.3.9removed forgotten pdb.set_trace() in revoke_pending_tasks()2.20v1.3.8cleanup2.21v1.3.7cleanup2.22v1.3.6log queue name2.23v1.3.5Readme updated2.24v1.3.4javascript helper viewsfix Task.set_progress(0)2.25v1.3.3make sure fields are unique in TaskAdmin fieldsets2.26v1.3.1unit tests verified with Python 2.7/3.6/3.7 and Django 1.10/2.02.27v1.3.0cleanupclassify as production/stable2.28v1.2.5Tested with Django 2.0 and Python 3.7Renameasynctois_asyncto support Python 3.7DJANGOTASK_REJECT_IF_NO_WORKER_ACTIVE_FOR_QUEUE app setting addedexample cleanup2.29v1.2.4API to create and run task via ajax2.30v1.2.3TaskAdmin: postpone autorun to response_add() to have M2M task parameters (if any) readyTask.clone() supports M2M parameters2.31v1.2.2property to change verbosity dinamically2.32v1.2.1util revoke_pending_tasks() added2.33v1.2.0DJANGOTASK_JOB_TRACE_ENABLED setting added to enable low level tracing in Job.run()Added missing import in utils.py2.34v1.1.3cleanup: remove get_child() method being Task an abstract classfix: skip Task model (being abstract) in dump_all_tasks and delete_all_tasks management commandsgeneric get_model_from_id() helperJob.on_complete() callback2.35v1.1.2provide list of pending and completed task status2.36v1.1.0INCOMPATIBLE CHANGE: Make model Task abstract for better listing performancesredundant migrations removedconvert request.body to string for Python3pretty print task params in log when task completes2.37v0.3.8return verbose name as description2.38v0.3.7description added to Task model2.39v0.3.6More fixes2.40v0.3.5log to field fix2.41v0.3.4log quickview + view2.42v0.3.3Optionally log to either file or text fieldManagement commands to dump and delete all tasks2.43v0.3.2search by task.id and task.job_id2.44v0.3.1Keep track of task mode (sync or async)2.45v0.3.0new class Job provided to share task-related logic among job funcs2.46v0.2.0fixes for django 2.x2.47v0.1.15hack for  prepopulated_fields2.48v0.1.14css fix2.49v0.1.13minor fixes2.50v0.1.12Deferred Task retrieval to avoid job vs. Task race conditionImproved Readme2.51v0.1.11superuser can view all tasks, while other users have access to their own tasks onlyjs fix2.52v0.1.10prevent task.failure_reason overflow2.53v0.1.9app settings2.54v0.1.8always start job from task.run() to prevent any possible race conditiontask.run(async) can now accept async=False2.55v0.1.7javascript: use POST to retrieve tasks state for UI update to prevent URL length limit exceed2.56v0.1.6Improved ui for TaskAdminFix unicode literals for Python32.57v0.1.5fixes for Django 1.10send_email management command example added2.58v0.1.4Fix OneToOneRel import for Django < 1.92.59v0.1.3Polymorphic behaviour or Task.get_child() restored2.60v0.1.2TaskCommand.run_task() renamed as TaskCommand.run_job()New TaskCommand.run_task() creates a Task, then runs it;
this guarantees that something is traced even when background job will fail