serial: fix port cleanup and use-after-free