serial_link: Give up on IOError in ListenerThread.

This prevents spinning and eating memory when the USB port is unplugged with pyserial.
pylibftdi will still fail.