implements sql tokenizer logic