added IndexBuilder and Splitter
added Aggregator and Sorter, changes at Splitter
added CountNormalizer, minor changes
added NGramSplitterWithCount and TypoSplitterWithCount (work in
progress)
small changes at Splitter.java
changed main method at IndexBuilder
changed testwiki to enwiki
small fix at IndexBuilder
changed index format and therefore added BinarySearch.java
index is smaller now
changed input wiki...
increased index size again
index has to be smaller...
fixed bug caused by String concatenation
some tests
Merge branch 'splitter' of https://github.com/renepickhardt/typology into splitter
included a very small example that demonstrates the power of lucenes FST implementation to answer scored prefix queries
optimized IndexBuilder, modified sorting and normalizing of 1grams and
0edges
added new DataSetSplitter (renamed old one to DataSetSplitterOld)
more tests of lucene fst on 5grams