Fix case where we have looked ahead when in infix mode, split a token on
a :, and then switch back to prefix mode and want not to have split the
token on the :