fixed issue where tokens with numeric values wouldn't be flagged as duplicate even with allowDuplicates set to false