Multiple thread, multiple data access web crawler.