nlplab/nersuite — pull request #14 by priancho (2012-07-02 08:05:47+00:00)
Title: Minor changes of -multidoc option, token offset calculation

For full history, please refer to README file

Version 1.2.1 (2012.07.02)
- Change: -multidoc [SEP] option for all programs now regards lines "beginning
with" [SEP] as comment lines and print them without any modification,
whereas it previously dealt with the lines "exactly same" to [SEP].
In addition, multiple comment lines can be used from this version.

- Change: nersuite_tokenizer computes token offsets at document level, whereas
it used sentence level token offsets. When it encounters a document
separator, which is given by -multidoc [SEP] option, the tokenizer
resets the offset to 0.

- New function: correct_mp program which removes named entities having mismatched
parenthesis is added. Documentation will be added to the homepage
later.