Remove epoll handlers when we close connections, and get rid of references to handlers on closures when we're about to free them