drivers: serial: revert a hack

Since we've correct the config, so revert this hack:
11ea3c25b18f6b982fc6bcbe32ea42102aa7a158