More sentence boundary tweaks.
Added updated generated tokenizer source code.