Tokenizer now creates tokens instead of strings.

Tokens are of type token_t and have a data part (std::string) and a
type part (enum { string, number, command, invalid }).