reimplemented lucene ingester to help with tf-idf vectors