alphagov/whitehall — pull request #479 by h-lame (2013-04-23 11:24:44+00:00)
Title: Rummage later

For: https://www.pivotaltracker.com/story/show/48395245

Turns out the problem was that publishing a policy with many (800+ for the docs mentioned in the bug) related editions was taking too long. The process is then killed before it completes (but it's not in a transaction so parts of it succeed) and retried on another server, which immediately responds with a lock version error.

To fix we've made Searchable#update_in_search_index and Searchable#remove_from_search_index create a job to do the actual communication with Rummager. This is faster; it went down from 40s to 10s on my dev machine in the console (e.g. so rummager communication is a no-op), but it's not fast enough (~10s is still slow and might be one of the timeout limits before it is retried). So we've also turned the reindex all the related editions for a policy (see ``PolicySearchIndexObserver``) into a delayed action. This made the process go back down to 500ms in the console on my machine.

Note: these jobs get added to a unique queue ``Whitehall.rummager_work_queue_name``, they should be picked up by normal workers, but it means we can scale the workers just for that queue if we need to.