add better polling and add support for scheduled jobs