finished batchification for all crawlers