Actually honour the serial port config