Only destroy the socket if it exists.