don't try to destroy the context again when in blocking mode
just reordering a bit the logic in the destructor

it doesn't really change anything it's just that the new sequence
is more clear to follow
align to last changes in the async api