Merge reusable portions of web crawl functionality