mozilla/kitsune — pull request #623 by willkg (2012-05-30 19:25:30+00:00)
Title: Parallificationylzing 720935

This covers two things:

* rewrite of indexing that allows it to run multiple indexing chunks in parallel in separate celery tasks when run from the admin
* reworks output for ES related commands from using log.info to print to be less spammy

The latter isn't very interesting. Figured it makes it easier to test, so I included it here.

The former has some things to think about when doing a review:

1. I tried to reduce the gap in time between when you click on the "reindex" button in the admin/search to when it starts indexing and the admin/search knows that it's indexing. I did that by having the handler for "reindex" do a lot of work. That work takes about 30 seconds on my machine. I don't know offhand what the page timeout value is, but if it's less than the time it takes to do the work, then that's a problem.

2. The new admin/search page tells you the number of outstanding indexing tasks at the top, but doesn't show progress for those tasks. I figured I'd see if this was good enough and if not, then I'd go implement the progress stuff as a separate bug. Having fiddled with it a few times, I think it's probably good enough and that with historical knowledge of how long indexing takes, we can guestimate the progress and ETA and that's "good enough".

3. This breaks the work up by model then by 50,000 item chunks. It runs all of these at the same time. We have some options here. I could break it up into smaller chunks. I didn't see an option for the ``@task`` decorator to limit how many it runs in parallel at a time. I don't want to flood celery in production with indexing tasks, but I'm not sure the way I broke up the tasks is ideal. I think this is a good first pass and we can fiddle with it later.

4. I kept the Record thing, but now it generates one record per indexing chunk. The problem here is that in an indexing pass, it'll generate 6 Records. This is a bit messy and it's not displayed well. I figured it's good enough for now and we could deal with fixing it later. Probably by adding a "batch_id" column and then grouping on that and showing the batches in groups. Anyhow, thoughts on that are useful, but I'm pretty sure unless they're dire, we should push the work off to a new bug.

I think that covers it.

r?