Updated multithreaded version with criteria for ending crawl