Release pool reference when closed, and don't blow up if the socket is already closed.