Wikia/app — pull request #703 by relwell (2013-04-24 22:30:34+00:00)
Title: Major Search Querying Update

This update includes several important changes that result in a major increase in search relevance.

This is accomplished in part by the following changes:
* Removal of Nested Queries
* Inclusion of Backlinks as a Query Field
* Removal of Boost Queries from On-Wiki Search
* Re-Tweaking of Query Field Boosts

Removal of Nested Queries
----------------------------------------
Our original approach for searching with Solr as of 2/1/2012 was a DisMax query that scored all documents matching the phrase and then filtered out documents that did not match the provided wiki ID. In order to address our needs for robust matching as well as wiki filtering, we had to move to using Lucene query syntax for a number of statements required to minimize scope, and then conjoin with a DisMax nested query to operate against a minimum of candidate documents. This introduced complexity and weakened performance.

The ExtendedDisMax parser, introduced in Solr 3.1, lets us provide robust matching for user input while allowing for full Lucene query syntax. This makes our queries much easier to read, and puts many of the parameters passed to the subquery as variables in the GET request, where they belong. In the future, we will also be able to experiment with bigram and trigram matching for improved relevance, which is something that this parser supports off the shelf.

Inclusion of Backlinks as a Query Field
--------------------------------
Backlinks have now been introduced as a field against which we are querying using the ExtendedDisMax parser. Backlinks are populated by storing outbound link text and the outbound link's document id for a given document, and then "inverting the graph", using a backend process, storing the inbound link text for a given document in a separate field. This has been tested as working correctly against about 30 wikis, generally among our "use case" wikis. That is to say, the majority of our wikis do not yet have this feature, but those that do have it show a clear improvement upon querying such a field. Backlinks are likely to be introduced to more wikis upon a future full site reindex.

Removal of Boost Queries from On-Wiki Search
------------------------------------
One of the side effects of using nested queries for our dismax capabilities was the need to add boost queries to place a higher preference on proximity matches. Removing this feature actually gives us better results, based on our tests.

Re-Tweaking of Query Field Boosts
-----------------------------------------
With backlinks included as a query field, we used this opportunity to experimentally improve our search relevance based on 122 different use cases populated by the community. The arrived-upon values were found to provide the best score among a number of different configurations.