Don't add duplicate blacklisted/whitelisted users