Refactor handling of poller workers