Move poller initialization retry logic from within poller to Work()