Fix Tokenizer Reader field to be correctly initialized
add some docs
remove normalizecharreader: it screws up the viterbi costs, add a filter instead. if you really want to do this before tokenization, you can use a charfilter
don't create useless object
clear warnings
make the code 1.5 compatible
add part of speech tag documentation
some tuning of defaults stopTags, and custom attribute reflection
improved docs and attribute reflection for POS and pronunciations
bugfixes/tests to the analyzer, support protected words in BasicFormFilter/KatakanaStemFilter
add random testing and fix a bug found where streamtagger would crash on certain inputs
add a random test for the analyzer too
add example configuration
merge r34
add prettify
add trivial unit test for this filter
add CostAttribute that displays viterbi cost
add another basic unit test
set the abs cost for the attribute
cost values reset for new sentences, so expose a boolean in Token so we know when a new sentence begins. we could also use this in the future e.g. sentenceIncrement