refactoring the crawlers.