Selectively clear restriction cache for whitelisting