Fix server close issues.

Server will not close until all sockets are destroyed.