Temporary work to develop a proper vectorizer for twitter data

Basically use the twitter tokenizers to delimit words, hash-tags,
addressees, stock-tickers and emoticons, then (work yet to be done)
wrap that in a lucene adaptor, and use the lucene tools to subsequently
lower-case the text, remove stop-words, and stem the remaining words.