[WIP] Changes to crawler API

open discussion.