alphagov/rummager — pull request #245 by rboulton (2014-06-18 14:27:24+00:00)
Title: Add popularity info when bulk loading

When bulk loading from a dump (eg, from whitehall), popularity
information needs to be added to the loaded documents. Similarly, when
loading a dump of best-bets from search-admin, a special
`stemmed_query_as_term` field needs to be added. This wasn't previously
being done, because the documents being loaded in weren't being parsed
and were just being passed straight to elasticsearch.

If we want to load without parsing, we can do that by making a call to
upload a dump file to elasticsearch directly using a tool like curl (and
more efficiently than going through ruby). The normal case for loading
is that we want to import a dump from whitehall, or from search-admin.

There's a lot of technical debt in this indexing code, but I'm just
ignoring it for now because when we implement an indexing worker
listening to messages from the content store we'll want to rewrite the
indexing pathway (and can thus address the messiness then).

Related story: https://www.pivotaltracker.com/story/show/73289170