removed boost/tokenizer since its not used, and refactored code to make it more obvious