modify compiler using Naist jdic for ChaSen
add a chance to specify custom dictionaries to be compiled
modify build.xml to accept naist jdic for chasen
just arrange indent. I just wanted to do it before adding dist target etc.
add ant description and fix naist-chasen.properties
add dist target
add use of CompositeTokenFilter
change version 1.0-dev to 1.1-dev
JapaneseBasicFormFilter can throw NPE when it is used with WordDelimiterFilter.
add clean-sen target
issue#5: fix incorrect highlighting, the tagger did not correctly accumulate buffer offsets and sentence offsets
issue#6: on linux, SentenceTest would fail because the files were not necessarily processed in a certain order (so the possible POSs could come back in arbitrary order).
do not say BUILD SUCCESSFUL if tests fail
issue#7: compiled jar doesn't include resources (stopwords, stoptags, etc)
add CHANGES.txt entry
upgrade to lucene 3.2.0 jar files
use BaseTokenStreamTestCase.checkRandomData(), extend LuceneTestCase in all test classes
performance: types do not change across case pairs
performance: don't lookup type unless we actually need it (basic latin and katakana only)
performance: swap order of testing