progression to crawler service and various additions to facilitate required functionality