improve performance for large datasets where url duplicates need not to be checked