process input as tokens