refactor crawlers to be more generic