updated duplicates list taking both blacklisting/whitelisting into account.