moses-smt/mosesdecoder — pull request #44 by jgwinnup (2013-08-16 18:44:03+00:00)
Title: Fixed bug in tokenizer.perl where comma separated lists of single characters aren't handled correctly, updated Russian NBP list

Committing fixes from a colleague not on GitHub - minor tokenizer fix and updated Russian nonbreaking prefix list

Fixed bug in tokenizer.perl where comma separated lists of single characters aren't handled correctly

input> A,B,C,D,E,F

yielded> A, B,C , D,E , F

now yields> A, B, C, D, E, F

Updated Russian nonbreaking prefixes list with capital letters