duplicates removal and external debug flag