Don't try to disconnect if we never connected in the first place

If we do that means we pass a NULL pointer into the disconnect function,
which it (currently) asserts will not happen.