Converted the tokenizer setup to be more canonical C++11.