speed and memory improvements to duplicate checks

even when there are duplicate tokens in the tokenlist, we now filter
most of them out during generation before the final dupcheck
(even with --no-dupchecks enabled); this results in significant speed
and memory improvements with certain tokenlists