Close socket in any case.