mozilla/kitsune — pull request #438 by willkg (2012-01-17 22:02:08+00:00)
Title: [bug 715932] Make ES indexing less sucky

* The ES connection already has code for forcing bulk, so we don't need to
repeat that. this changes the code to push the setting to ES.
* This also tweaks the estimation code so that it shows minutes and seconds and
shows the total delta later. Now I can stop running
"time ./manage.py esreindex".
* Fix esreindex so that you can specify doctypes. This will appropriately
create/delete indexes so that what you don't want to delete won't get
deleted.
* Adds basic handling for bad data.

This does a log.exception, but we really should log more than that and/or
make it more obvious to developers that there's bad data out there.

In the meantime, this allows us to continue indexing.
* Reduced memory usage of indexing by iterating over ids---now it runs on
my laptop.
* Changes _get_index() to get_es_index(). We use it so often it might as
well be part of the "public API".
* Fixed create/delete indexes so that to switch doctypes to their own index
is now just a change in settings---no code changes needed.

End result of this is that indexing doesn't die if it hits bad data, indexing
takes less memory to run, you can specify specific doctypes to index at the
command line, and the code is a little better.

r?