Make blacklist actually work when notify is the configured action