refactor crawl and add tests