Convert lexer to C++.

In the process, fix some embarassing usage of char* strings.  This is
why I prefer std::string, thank you very much.