Check socket and ssl connection exist before closing.

As the socket or the ssl connection might not be initialized when
closing, we need to check its existance before doing it.