Using a hash for duplicate lookup

As @mgruner suggested it's much more performant to use a hash for checking for duplicates.