refactoring the crawl code