Yet another iteration. Just prctl pre-setuid to inherit (full) root capabilities to nobody, and then remove all capabilities except what we want.
Added SO_BINDTODEVICE to IPv4 path too.