awesomo/django-ztaskq — pull request #7 by simonedeponti (2012-08-30 16:56:22+00:00)
Title: Added delayed task support

Hi,

I have added delayed task support to `django_ztaskq`. This was needed because I wrote a Django email backend, [django-ztaskq-mailer](https://github.com/abstract-open-solutions/django-ztaskq-mailer), that needs to retry sending a mail "some time later" when it encounters an error.

From the "user" perspective, very few has changed. The most important difference is that now you can do:

```python
mytask.async('eggs', with='spam', and='spam', ztaskq_delay=30)
```

And your task will not be executed before the 30 seconds are expired (barred a few milliseconds difference).

In order to do this I had to refactor code a bit. I'll provide you with a quick summary of changes and their reason:

* I use `pylint` to check my code within the editor as I type, so I have "pylinted" it
* I updated the version number to 0.3.0. This is needed so I can correctly pin the dependency on my async mail backend
* I removed the use of the autoreloader, because correctly initialing and properly closing `zmq` sockets is hard enough by itself and I did not want to add the problem of having two processes running each time I launched a service (and one of the two actually happened to stay around because the thread containing the `zmq.core.device.device` just wouldn't die). In development, restarting the worker should be enough in most cases and will save you from strange `zmq` errors
* I reworked the logger a bit, so that it tries to use a separate log file for each process when possible. This forced me to add an extra, required arg to the `workerd` call, which is the name of the worker. This way one can determine on which worker happened what. Furthermore, if the path of `ZTASKD_LOG_PATH` contains `%(name)s`, the name of the process is substutituted there (so you son't have two processes fighting for the log file)
* The file logger is now a `RotatingFileHandler`, so you don't need logrotate to mess with your daemons.
* I now use `inproc://` for the `ztaskd` internal queue, so that any error that might arise from insufficient permissions on `/tmp` (as was the case with `ipc://` that uses a UNIX socket) can be averted
* I use the `atexit` module to do the shut-down cleanup
* Since Django 1.4 should have properly-timezoned timestamps passed to its models, I now use `pytz` to mark timestamps correctly as being in UTC
* I refactored the internal code of `ztaskd`, which now works slightly different: each time the server gets a new task, it spawns a thread to handle it (that is, send it to the worker via the internal "bouncer"). This thread is a normal thread that executes very fast and dies off immediately if it is not a delayed task, while if it is a delayed task the thread "sleeps" until the time to pass it to the worker has come. Also a new routine has been added to requeue eventual "left-overs" from a previous run. The only thing that's a bit shaky in it is probably the finished threads "garbage collection", but I think it works well enough for most cases.

That said, there are two big warnings to be aware of: due to my changes, this code is unlikely to work on anything that has a version of `pyzmq` older than 2.2.0 or any Django release prior to 1.4.

Thanks a lot for this piece of software!