Improve the setStandardBaudRate() on Linux

* The ASYNC_SPD_CUST flag shall be cleared only if it was earlier
set; otherwise it is not necessary to touch of the serial struct.

* Seems, that is more reasonable to return an error code when
writing back of the serial struct is failed. In this case it is
an error, because we know that serial struct is supported, since
we got it successfully earlier.

Tested build on Archlinux 64 bit using Qt4 and then Qt5.
Tested build on Android x86 using Qt5.

Change-Id: I54ca4772dc770dabac5e3555e6cb42ecc23255a4
Reviewed-by: Rafael Roquetto <rafael.roquetto@kdab.com>
Reviewed-by: Sergey Belyashov <Sergey.Belyashov@gmail.com>