Changed tokeniser to handle words as soon as available (i.e. handling blocked i/o calls).