Lexical analysis (clean version)

Process of converting a sequence of characters into a sequence of
tokens. The majority of the code is translated into English.