Fix TcpListener frequently not accepting new connections

Small fix for a bug we stumbled over in spray