Fix race condition causing notification to reappear after being dismissed

Signed-off-by: mvglasow <michael -at- vonglasow.com>