Fixed erasing whitelists when changing restrictions

Fixes #1499