decide rule or assignment based on symbol terminating the statement; start tokenizing as assignment but if wrong decision, retokenize as a rule