Complete implementation of crawling