scan the input stream and generate the token stream