witgo/CRF — pull request #2 by MRudolph (2014-10-13 14:10:01+00:00)
Title: Feature/optional lowercase

This pull request extends the command line tool for training and evaluation (iitb.Segment.Segment) by making downcasing of tokens optional.

This seems to be a destructive action, because it's done before the features are generated.
Some languages (e.g. german) depend on capitalisation for distinguishing words, so this might be a valuable resource which should not removed.

For not breaking existing setups, there are new methods which can handle the optional downcasing.
It's on by default, but can switched off by adding "lowercase=false" to the configuration.

Tests are included and succeed (they are modified copies of the tests for the original tests).

Running the applications with the sample dataset also seems to work fine.