complete duplication checking