Fix memory leak

Remove now unused file descriptor list build by init_listeners().