Fix the memory leak in the RpcServer by closing connected sockets.