rubygems/rubygems.org — pull request #455 by karmi (2012-08-30 18:01:57+00:00)
Title: Improved search feature (elasticsearch based, demo available)

This pull request contains a proposed search feature overhaul for Rubygems.org, implemented with the [elasticsearch](http://elasticsearch.org) search engine, via the [Tire](https://github.com/karmi/tire) library.

### Objectives ###

The main objective of the effort is to allow searching in more gem properties then just their _names_, notably in _summaries_, _descriptions_ and _authors_ – technically speaking, to increase both [precision and recall](http://en.wikipedia.org/wiki/Precision_and_recall) of search at Rubygems.org.

Using a search engine — as opposed to a `LIKE %term%` database query — allows not only for better, faster searches, but also for advanced features such as a rich search query language, faceted navigation, and more.

### Changelog ###

All the steps required for implementing the feature are commited on the [`search-steps`](https://github.com/karmi/rubygems.org/compare/search-steps) branch, with extensive commit messages documenting the process. The important steps are:

* [`86240f2`](https://github.com/karmi/rubygems.org/commit/86240f2) and [`3a17730`](https://github.com/karmi/rubygems.org/commit/3a17730) implement the most simple search with elasticsearch, adding model integration and using Tire in the controller

* [`fa7d1cd`](https://github.com/karmi/rubygems.org/commit/fa7d1cd) adds complex mapping definition for the Rubygem model, allowing to search in gem summaries/descriptions, authors, dependencies, and more.

* [`8c6bee8`](https://github.com/karmi/rubygems.org/commit/8c6bee8) adds a sliding panel to the search results page which contains examples of searches with [Lucene search query syntax](http://lucene.apache.org/core/3_6_1/queryparsersyntax.html).

Additional Cucumber scenarios were added to document the new search features. Some [additional tweaks](https://github.com/karmi/rubygems.org/compare/karmi:search-steps...karmi:search-travis) were required to run the test suite successfuly at Travis CI.

Please review the [branch compare page](https://github.com/karmi/rubygems.org/compare/search-steps) to see the full picture.

### Demo Application ###

A demo application is available at ****.

Try out simple searches such as [`rack`](http://rubygems-with-elasticsearch.herokuapp.com/search?query=rack) or searching in authors: [`author:john`](http://rubygems-with-elasticsearch.herokuapp.com/search?query=author%3Ajohn) and dependencies: [`uses:rack`](http://rubygems-with-elasticsearch.herokuapp.com/search?query=uses%3Arack). More tips are available as [in-page help](http://rubygems-with-elasticsearch.herokuapp.com/search#tips).

The database contains only a limited subset of gems. The application is running on a free [Heroku](http://www.heroku.com) plan. The elasticsearch service is running on a Amazon EC2 `t1.micro` instance. Keep in mind, that the application runs in a tweaked development mode (due to issues with assets etc.), so the demo application performance does not reflect the performance in the real production environment.

If a dump of the Rubygems production database would be available, I'd like to import it into the demo application database.

### Further Development ###

If the proposed search implementation is considered desirable, a number of further developments is possible, eg.:

* more fine-grained score computation based on number of downloads, not straight sorting,
* highlighting the relevant matched snippets from gem properties,
* adding faceted search on authors,
* linking to a specific matched version from search results,
* displaying aggregated statistics such as authors with most gems, authors with most downloaded gems, etc.,
* adding Tire's [NewRelic instrumentation](https://github.com/karmi/tire-contrib/blob/master/lib/tire/new_relic/instrumentation.rb) to track performance.

### Installation Instructions ###

To check out the search feature locally, assuming you had cloned the Rubygems.org repository, set it up according to [instructions](https://github.com/rubygems/rubygems.org/wiki/Development-Setup) first:

./script/setup

To import your local gems into the database, run:

bundle exec rake gemcutter:import:process

Then, [install elasticsearch](http://www.elasticsearch.org/guide/reference/setup/installation.html) using your preffered method. On Mac OS X, the easiest way is to use Homebrew:

brew install elasticsearch

To import the gems from the database into elasticsearch, run:

bundle exec rake environment tire:import CLASS='Rubygem' FORCE=1