tokenized input
use Java's built-in lexigraphical analyzer