Tokenizer complete+added ability to support circular buffers.