update diff
add statistics about error distribution in the corpus
adding comments about how f measure changes when we prefer text version
small performance improvement
update Lucene version
major indexing speedup (factor 2-3)
properly close resources