This token value is written into the 'current token'. But that's not enough to tokenize. So, we then make a few decisions based on what the token was. For most of the operators, all necessary scanning was done with just that one non-alphanumeric character, so all they have to do is advance the list pointer to the next token. A few other operators might have more characters, so they check the next one and advance the stream if neccessary.
Assignment is special.
Of course, if there isn't a variable before it, then there's definately a syntax error in the string. For numbers, we have to do a quite a bit of work. For decimal numbers, if we finish over a decimal point, we can turn the number into a real value and start reading numbers after the decimal point. We try to see if its a function be comparing it to all the names in the function table. Otherwise, we can assume it's a variable, and thus mark the start and end of the name for use later on.