Introduce isolated poller workers

This is breaking change from user API point of view.