Duh, make the socket non-blocking