Guard against no subscription in on_unsubscribe

Don't assume subscribe_to has been called
on a subscriber when we're unsubscribing it.