Package: django-gearman-jbox
django-gearman-jbox===================*django-gearman-jbox* is a convenience wrapper for the [Gearman][Gearman][Python Bindings][python-gearman].With django-gearman-jbox, you can code workers as well as clients in a Django projectwith minimal overhead in your application. Server connections etc. all takeplace in django-gearman-jbox and don't unnecessarily clog your application code.This library is based in large part on Fred Wenzel's [django-gearman] and Jozef Ševčík's [django-gearman-commands].- [django-gearman][django-gearman] for the 'decorator way' to create workers (very cool)- [django-gearman-commands][django-gearman-commands] for the 'gearman_server_info' command (very nice)But there are some modifications :Workers are now launched individually, so you have to pass 2 mandatory parameters to start the worker :- the Django App name where workers reside with the `-a` parameter- the worker's name with `-n`The `-q` parameter is still here and has the same function than in [django-gearman].- I removed the ability of lauching many wokers at once (`-w` parameter in [django-gearman]).I prefer having one process for each worker and Supervisord managing process. (screen is not an option)- I also added a signal handler to catch SIGTERM signals send by Supervisord and SIGINT when you worker is attached to the console.This gives you the possibility of executing code just before the worker terminates.(See `django_gearman_jbox\management\commands\gearman_worker.py`, line 116)[Gearman]: http://gearman.org[python-gearman]: http://github.com/samuel/python-gearman[django-gearman]: https://github.com/fwenzel/django-gearman[django-gearman-commands]: https://github.com/CodeScaleInc/django-gearman-commandsInstallation------------It's the same for both the client and worker instances of your django project :$ pip install django-gearman-jboxAdd ``django_gearman_jbox`` to the `INSTALLED_APPS` section of `settings.py`.Specify the following setting in your local settings.py file:# One or more gearman serversGEARMAN_CLIENT_SERVERS = ['127.0.0.1']GEARMAN_WORKER_SERVERS = ['127.0.0.1']Workers-------### Registering workersCreate a directory `gearman_workers` in any of your django apps, and define as manyworkers as you like, one worker per file. Create an empty `__init__.py` so the directory will beloaded as a package.Example :my_django_app|_ models.py|_ gearman_workers|_ __init__.py|_ worker_foo.py|_ worker_bar.py### Registering tasksIn the worker file, you can define as many tasks as functions as you like.The function must accept a single argument as passed by the caller and mustreturn the result of the operation, if applicable. (Note : It must accept an argument, even if you don't use it).Mark each of these functions as gearman tasks by decorating them with :import django_gearman_jbox.decorators.gearman_task@gearman_task()def my_task_function(foo):pass### Task namingThe tasks are given a default name of their import path, with the phrase`gearman_task` stripped out of them, for readability reasons. You can overridethe task name by specifying `name` parameter of the decorator. Here's how :import django_gearman_jbox.decorators.gearman_task@gearman_task(name='my-task-name')def my_task_function(foo):pass### Task parametersThe gearman docs specify that the task function can accept only one parameter(usually refered to as the ``data`` parameter). Additionally, that parametermay only be a string. Sometimes that may not be enough. What if you would liketo pass an array or a dict? You would need to serialize and deserialize them.Fortunately, django-gearman-jbox can take care of this, so that you can spendall of your time on coding the actual task.@gearman_task(name='my-task-name')def my_task_function(foo):passclient.submit_job('my-task-name', {'foo': 'becomes', 'this': 'dict'})client.submit_job('my-task-name', Decimal(1.0))### Tasks with more than one parameterYou can pass as many arguments as you want, of whatever (serializable) typeyou like. Here's an example job definition :@gearman_task(name='my-task-name')def my_task_function(one, two, three):passYou can execute this function in two different ways :client.submit_job('my-task-name', one=1, two=2, three=3)client.submit_job('my-task-name', args=[1, 2, 3])Unfortunately, executing it like this:client.submit_job('my-task-name', 1, 2, 3)would produce the error, because ``submit_job`` from Gearman's Python bindingscontains __a lot__ of arguments and it's much easier to specify them viakeyword names or a special ``args`` keyword than to type something like seven``None``s instead :client.submit_job('my-task-name', None, None, None, None, None, None, None, 1, 2, 3)The only limitation that you have are gearman reserved keyword parameters. As ofGearman 2.0.2 these are :* data* unique* priority* background* wait_until_complete* max_retries* poll_timeoutSo, if you want your task definition to have, for example, ``unique`` or``background`` keyword parameters, you need to execute the task in a special,more verbose way. Here's an example of such a task and its execution :@gearman_task(name='my-task-name')def my_task_function(background, unique):passclient.submit_job('my-task-name', kwargs={"background": True, "unique": False})client.submit_job('my-task-name', args=[True, False])Finally:client.submit_job('my-task-name', background=True, unique=True, kwargs={"background": False, "unique": False})Don't panic, your task is safe! That's because you're using ``kwargs``directly. Therefore, Gearman's bindings would receive ``True`` for``submit_job`` function, while your task would receive ``False``.Always remember to double-check your parameter names with the reserved wordslist.### Starting a workerTo start a worker, run `python manage.py gearman_worker -a <django_app_name> -n <worker_name>`. It will startserving all registered tasks for that worker.Example :$ python manage.py gearman_worker -a django_app_name -n worker_foo$ python manage.py gearman_worker -a django_app_name -n worker_barTo spawn more than one worker see Supervisord configuration below.### Task queuesQueues are a virtual abstraction layer built on top of gearman tasks. Aneasy way to describe it is the following example: Imagine you have a taskfor fetching e-mails from the server, another task for sending the emailsand one more task for sending SMS via an SMS gateway. A problem you mayencounter is that the email fetching tasks may effectively "block" the worker(there could be so many of them, it could be so time-consuming, that no othertask would be able to pass through). Of course, one solution would be to addmore workers (via the Supervisord), but that would only temporarilysolve the problem. This is where queues come in.The first thing to do is to pass a queue name into the job description, likethis :@gearman_task(name="task_foo", queue="foo")def function_foo(some_arg):pass@gearman_task(name="task_bar", queue="bar")def function_bar(some_arg):pass@gearman_task(name="task_babar", queue="bar")def function_babar(some_arg):passYou may then proceed to start the tasks that are bound to a specificqueue :python manage.py gearman_worker -a <django_app_name> -n <worker_name> -q barBe aware of the fact that if you don't specify the queue name, the workerwill load all tasks.### Start workers with SupervisordSupervisor - http://supervisord.org/ is babysitter for processes.It allows you to launch, restart and monitor running processes. In our case it will be workers.To do so, create one config file by worker and adjust the number of workers you want with the 'numprocs' parameter :`worker_foo.conf` :[program:worker_foo]command         = /path-to-your-virtualenv/bin/python /path-to-your-project/manage.py gearman_worker -a <django_app_name> -n %(program_name)sprocess_name    = %(program_name)s_%(process_num)02dnumprocs        = 1autostart       = trueautorestart     = trueuser            = myappdirectory       = /home/myapp/environment     = HOME='/home/myapp',USER='myapp',LOGNAME='myapp',`worker_bar.conf` :[program:worker_bar]command         = /path-to-your-virtualenv/bin/python /path-to-your-project/manage.py gearman_worker -a <django_app_name> -n %(program_name)s -q barprocess_name    = %(program_name)s_%(process_num)02dnumprocs        = 2autostart       = trueautorestart     = trueuser            = myappdirectory       = /home/myapp/environment     = HOME='/home/myapp',USER='myapp',LOGNAME='myapp',You can also create a `groups.conf` file with this content :[group:foo]programs=worker_foo, worker_foo2[group:bar]programs=worker_bar, worker_bar2This will create process 'group' and allows you to reload all workers related to this group at once when you redeploy new code.Once you're config files are created, do `/etc/init.d/supervisord start` to start Supervisord and `supervisorctl reload` if you modify config orsupervisorctl rereadsupervisorctl updatesupervisorctl restart foo:*supervisorctl restart bar:*### Execute code when workers dieWorkers catch SIGTERM and SIGINT signals to kill themselves with a `sys.exit(0)` in a callback function.At this point in the code you can add your own function(s) which will be executed before the `sys.exit(0)`See `django_gearman_jbox\management\commands\gearman_worker.py`, line 116Note that this will impact all workers as it resides in the `gearman_worker.py` script which is global for all workers.Clients-------To make your workers work, you need a client app passing data to them.Create and instance of the `django_gearman_jbox.GearmanClient` class and execute submit_job with it :from django_gearman_jbox import GearmanClientsentence = "The quick brown fox jumps over the lazy dog."client = GearmanClient()res = client.submit_job("foo", kwargs={"sentence": sentence})print "Result: '%s'" % resDispatching a background event without waiting for the result is easy as well :client.submit_job("foo", background=True, kwargs={"sentence": sentence})Gearman Server Infos-------------------`python manage.py gearman_server_info` outputs current status of Gearman servers.If you installed Prettytable dependency, here is how output looks like :$ python manage.py gearman_server_info+---------------------+------------------------+| Gearman Server Host | Gearman Server Version |+---------------------+------------------------+|    127.0.0.1:4730   |        OK 0.29         |+---------------------+------------------------+.+---------------+---------------+--------------+-------------+|   Task Name   | Total Workers | Running Jobs | Queued Jobs |+---------------+---------------+--------------+-------------+| data_unlock   |       1       |      0       |      0      || data_import   |       1       |      1       |      0      || cache_cleanup |       1       |      0       |      0      |+---------------+---------------+--------------+-------------+.+-----------+------------------+-----------+-----------------+| Worker IP | Registered Tasks | Client ID | File Descriptor |+-----------+------------------+-----------+-----------------+| 127.0.0.1 |   data_unlock    |     -     |        35       || 127.0.0.1 |   data_import    |     -     |        36       || 127.0.0.1 |  cache_cleanup   |     -     |        37       |+-----------+------------------+-----------+-----------------+If you have a lot of workers, you can filter output using command argument (case-sensitive):$ python manage.py gearman_server_info cleanup+---------------------+------------------------+--------------------+| Gearman Server Host | Gearman Server Version | Ping Response Time |+---------------------+------------------------+--------------------+|    127.0.0.1:4730   |        OK 1.1.3        | 0.0006871223449707 |+---------------------+------------------------+--------------------+.+---------------+---------------+--------------+-------------+|   Task Name   | Total Workers | Running Jobs | Queued Jobs |+---------------+---------------+--------------+-------------+| cache_cleanup |       1       |      0       |      0      |+---------------+---------------+--------------+-------------+.+-----------+------------------+-----------+-----------------+| Worker IP | Registered Tasks | Client ID | File Descriptor |+-----------+------------------+-----------+-----------------+| 127.0.0.1 |  cache_cleanup   |     -     |        37       |+-----------+------------------+-----------+-----------------+Licensing---------This software is licensed under the [Mozilla Tri-License][MPL]:***** BEGIN LICENSE BLOCK *****Version: MPL 1.1/GPL 2.0/LGPL 2.1The contents of this file are subject to the Mozilla Public License Version1.1 (the "License"); you may not use this file except in compliance withthe License. You may obtain a copy of the License athttp://www.mozilla.org/MPL/Software distributed under the License is distributed on an "AS IS" basis,WITHOUT WARRANTY OF ANY KIND, either express or implied. See the Licensefor the specific language governing rights and limitations under theLicense.The Original Code is django-gearman.The Initial Developer of the Original Code is Mozilla.Portions created by the Initial Developer are Copyright (C) 2010the Initial Developer. All Rights Reserved.Contributor(s):Frederic Wenzel <fwenzel@mozilla.com>>Jeff Balogh <me@jeffbalogh.org>Jonas <jvp@jonasundderwolf.de>Jozef Ševčík <sevcik@codescale.net>Nicolas Rodriguez <nrodriguez@jbox-web.com>Alternatively, the contents of this file may be used under the terms ofeither the GNU General Public License Version 2 or later (the "GPL"), orthe GNU Lesser General Public License Version 2.1 or later (the "LGPL"),in which case the provisions of the GPL or the LGPL are applicable insteadof those above. If you wish to allow use of your version of this file onlyunder the terms of either the GPL or the LGPL, and not to allow others touse your version of this file under the terms of the MPL, indicate yourdecision by deleting the provisions above and replace them with the noticeand other provisions required by the GPL or the LGPL. If you do not deletethe provisions above, a recipient may use your version of this file underthe terms of any one of the MPL, the GPL or the LGPL.***** END LICENSE BLOCK *****[MPL]: http://www.mozilla.org/MPL/