Join all poller_threads on close