OpenTreeOfLife/opentree — pull request #249 by mtholder (2014-04-02 19:55:21+00:00)
Title: Api on celery

We'll get the best user experience from the curator app if changes are pushed to github with each commit that merges to the master branch (that will trigger updating of the oti index, thanks the hooks that Jim installed). But we did not want the PUT verb for each study edit to be sluggish.

So, I've implemented an asynchronous call to the 'push to github controller' right before the return of a response from a successful POST, PUT, or DELETE verb in the API. Now the response will return, and that will trigger a call to the push controller in the API to send the changes to github, and the curator won't have to wait for that to complete.

This is on a branch the celery-push of the api [1].

Unfortunately, web2py is somewhat limited in terms of support for this kind of thing, and does not want us to spawn a thread and let the request return [2]. So we can't just do the easy thing and use a python thread.

So, the solution seemed to be (a) become more reliant on web2py-specific solutions, or (b) use an external deferred task management solution.

I went with solution (b)*. celery [3] seems to be the solution of choice, and it needs a broker for the deferred task. Redis is a common choice, and we have code to use redis for caching. So that seemed reasonable.

So, the celery-push branch of api needs redis and celery to be installed and configured. The code for this is on the 'api-on-celery' branch of the deployment system[4].

This is running on ot7. If the code on the opentree branch looks good to @jar, then we can merge both branches. But we can't merge the celery-push branch until the deployment branch is merged.

I'll copy this to the pull request...

cheers,
Mark

[1] https://github.com/OpenTreeOfLife/api.opentreeoflife.org/tree/celery-push
[2] https://groups.google.com/d/msg/web2py/qZ_RUAfqaOY/M6hm4CVeoyUJ
[3] http://docs.celeryproject.org/en/latest/
[4] https://github.com/OpenTreeOfLife/opentree/tree/api-on-celery