nvie/rq — pull request #69 by svetlyak40wt (2012-05-17 20:57:15+00:00)
Title: Preload job's code before the fork.

This way we will get significant performance boost if there are
many imports in the function's module.

For example, I created a `jobs` submodule in my flask based project and added a simple job like this:

def do_the_work(filename):
with open(filename, 'a') as f:
f.write('blah minor\n')

I found, that it takes about 30 seconds to perform 100 such jobs. It is 3 jobs per second only.

The reason is simple, worker import the module with function after the fork, and does it every time.

In this pull request, I've modified a code and added a `job.func` right before the fork, to bring all modules into the memory before the forks.

This way, worker was able to process the same 100 jobs in one second. 30 times boost!!!

*Probably*, it is better to remove the Job.func property and load function right in the Job.__init__. If you want, I'll modify by branch accordingly.