Completed tokenizer's top level nextToken method.