initial support for language detection and stemming
initial support for English lemmatization
change handling of HTML extraction to be more permissive
language detection in XmlExtractor now uses xml parser (instead of
autodetect parser)
better support for inputFormat at the extraction stage
more morphological analysis goodness
preliminary tool architecture
cleaning up tool architecture
better testing of json for document storer
- string storage and retrieval
- document storer and expander now used stored ids
adding strings storage and retrieval methods
preliminary tools for document loading
corpus building as one big document
note to self about building corpus by reading analyzed fields
some lucene analytics
massive changes (better late than never for committing)
SpanQueryParser now returns a map of labels and spans
more functionality for DocumentTermFrequencies query
simplified DocumentTermFrequencies for all terms
progress on corpus terms