serial: Fix up host timing

The USB device was triggering bugs in the Linux USB stack.  Possibly
a queueing issue, I'm not sure.  The end result was corruption in audio
playback, probably due to DMA timing issues.

Signed-off-by: Sean Cross <xobs@kosagi.com>