Guard against no subscription in on_unsubscribe

Don't assume subscribe_to has been called
on a subscriber when we're unsubscribing it.
Bit longer sleep for slow machines running messaging_spec