Empty whitelist results in *everything* being filtered out

The expected behavior for an empty whitelist is that the whitelist does nothing, and only the blacklist is used.

What actually happens is EVERYTHING becomes blacklisted.

This patch fixes that issue.