Fix that dis-/enabling notifications affect active checks

fixes #5500