Wikia/app — pull request #277 by relwell (2012-12-19 23:58:35+00:00)
Title: Refactor WikiaSearchIndexer

WikiaSearchIndexer has been refactored in the following ways:

* WikiaSearchIndexer::getPage has been refactored to use the MediaWIki API's action=parse logic, removing some hackish approaches to accessing article content
* Additional data from action=parse that had originally been accessed via the backend is included earlier in the pipeline (e.g. headings)
* Some logic in WikiaSearchIndexer::getPage has been refactored into separate methods for improved testability
* Logic handling video and image metadata has been refactored into its own method
* Some separate methods have been refactored to accommodate an array_merge() approach at integrating a full associative array

Let's not merge this without a bit of a discussion and scheduling, since this will require some additional changes to the [Solr worker script](https://github.com/Wikia/backend/blob/refactor_indexing/bin/solr-worker.pl) to be merged

These changes are motivated by an ultimate desire to move from a single monolithic pipeline for search indexing to a number of services making use of Solr's atomic update capability, available with Solr 4.0. This would allow us to asynchronously update a single field for a given document without retrieving or computing the rest of its field data.

Atomic update capabilities are desirable for scenarios where:
* a particular field value is consistent across a large set of documents
* a field value for a single document is updated more often than the document is, and that change does not trigger a reindex through our messaging system
* a field value is desired, but the process to compute it is too time-intensive for real-time event indexing