Added options to specify custom delimiters for multigrams, plus custom delims for test sets.  proper automatic tokenization for utf8 input.