building a smart web crawl