Prevent sending one final notification clear after disabling the setting

This just missed the merge of #147