utcompling/tshrdlu — pull request #2 by njwilson (2013-03-20 16:47:00+00:00)
Title: Support for indexing tweets (and other objects) with Lucene

Provides support for writing objects to a Lucene index and searching them. An index can be created in RAM, which is lost when the process exits, or can be created on disk and used from multiple processes.

To index tweets from the command line:

```bash
$ tshrdlu run tshrdlu.twitter.index.SafeEnglishStatusIndexer my-status-index
```

To search the index from the command line:

```bash
$ tshrdlu run tshrdlu.twitter.index.StatusSearcher my-status-index "good morning"
Searching index 'my-status-index': good morning

4.3832407: Good morning
2.1916203: Somebody tell me good morning :) don't everybody say it all at once !!!!
[...]
```

To write and read an index from code:

```scala
// Get index writer and reader
val indexName = "my-status-index"
val writer = StatusWriterFactory(indexName)
val reader = StatusReaderFactory(indexName)

// Add a tweet (assuming `newStatus` is a Twitter4J `Status` object)
writer.add(newStatus)

// Search for "morning" and print the top 10 status messages
reader.search("morning").foreach {
case (status, score) => println(score + ": " + status.getText)
}
```

For a longer description, build the documentation and view the `tshrdlu.twitter.index` package.

The functionality in `tshrdlu.twitter.index` is built on code in `tshrdlu.util.index` which makes it easy (I hope) to extend and index other types of objects.