Only populate the blacklist when the whitelist is empty