added an allow duplicates switch