bugfixing issue with tokenizer not properly flushing tokens