Simplify the attach_server stuff to 1 method, hold a reference to the socket.

Otherwise the FD could be closed by a GC.