serial_link: Give up on IOError in ListenerThread.

This prevents spinning and eating memory when the USB port is unplugged with pyserial.
pylibftdi will still fail.

<!---
@huboard:{"order":114.875,"milestone_order":145.0,"custom_state":""}
-->