race conditions fixes

This pull requests fixes race conditions in tests and listener (I've recently created issues in the main repository).
After this change I'm unable to get this test to fail if I intentionally break `server.Serve` to no longer be graceful.