nltk/nltk — pull request #808 by longdt219 (2014-12-10 07:27:14+00:00)
Title: Fix bug  Stanford tokenizer -- request to add -outputFormat inlineXML #735 

The list of valid options for string tokenizer is
https://github.com/stanfordnlp/CoreNLP/blob/master/src/edu/stanford/nlp/process/PTBTokenizer.java

Thus, for preserving the "newline character" we should use
==> st = StanfordTokenizer('stanford-postagger.jar', 'utf-8', java_options='-mx300m', options='-tokenizeNLs true')
==> st.tokenize(s)
[u'Good', u'muffins', u'cost', u'$', u'3.88', u'\*NL\*', u'in', u'New', u'York', u'.', u'Please', u'buy', u'me', u'\*NL\*', u'two', u'of', u'them', u'.', u'\*NL\*', u'Thanks', u'.']
The "\*NL\*" is the new line character.

Code modified accordingly to resolve this.