new crawlbot api. not backwards compatible any more.