Don't init UdpClient when server starts talking