Only abort Accept loop on EINVAL

The accept method call on the socket can return many different errors but you should only really abort if an `EINVAL` is returned.

I also added a defer to ensure that the socket is closed when the accept loop returns.