Blank lines in the input stream are ignored. Split full stops from the ends of words.
Abbreviations cause problems here.
This means that unrecognised abbreviations may cause the tokeniser to consider it to be the last word in the sentence meaning that it will split one sentence incorrectly in to two. Some morphological rules are applied to recognise abbreviations not previously seen. A further complication arises for the situation when a word is an abbreviation and the last word of a sentence. The tokeniser looks for this by looking at the next word. If it is capitalised then the current abbreviated word is assumed to be the last of the current sentence and the following capitalised word is taken to be the first word of the next sentence.