Added logic to short circuit on tokens that are unrecognized.