Return on errors in poller and worker

This allows for graceful exit when connection to redis has died.