Simple lexical analyzer. Rules can be defined in a style similar to lex.