Only notify 'in-thread' when notifications are actually enabled

This should fix #1371 by only notifying when actually enabled.