elasticsearch/elasticsearch-river-twitter — pull request #13 by gpstathis (2012-05-14 01:02:16+00:00)
Title: Support for indexing the entire twitter status JSON

Heya ;-)

The devs over here @Traackr participated in a Rackspace elasticsearch hackday back in March (http://elasticsearchhackday.eventbrite.com/). Our prototype was basic: use ES and the twitter river to look for tweets in a certain geographic area and by keyword and show the results in real-time on a map, augmented with a simplified version of the @Traackr scores (see project here git@github.com:Traackr/staalkr.git) .

One thing we realized while we were doing this is that the current incarnation of the twitter river is pulling a limited subset of the available data from the Twitter Stream API. For instance, we needed the number of followers to do our computations. So we added a one liner back during the hackday to make it work but I always wanted to come back to it and see if it can be done in a more complete fashion.

What we propose is the addition of a "full" flag in the twitter river configuration that will allow the onStatus method to pull the full response JSON into ES. This is a more expensive operation but it would at least make it available for those that opt to use it.

Check it out and let me know what you think!

-GS