Use accept4(2) on Linux.
Set client fd O_NONBLOCK.
Optimize set-fd-nonblock.