removes duplicate tokens before parsing
allow recognition of multi-character tokens at start of input