if there are no active workers, just destroy the packet
use preallocated client list, so we can be sure that we wont leak filedescriptors in case of multiple setup_listener calls