Revert "Fix whitelist empty settings caching"

This reverts commit 136c17bade289d78fb650ec3150d44bcd770a511.