On the pages that pass the third test we search for disqualifying words to keep pages out of our index. ISP, Internet service provider, adult content, etc. What is left is a selection of URL reports found by the spider that have been filtered to meet your requirements. While our Data base Builder spiders can find tens of thousands of targeted URLs per day, our World Spider can find millions of URLs per day and they can literally cover the whole Internet in a few months. One day of spidering can produce between 10,000 and 100.00 URLs depending upon the degree of filtering we do and whether we are spidering known URLs or searching for them first then spidering them.
Larger spiders can find millions of web sites per day.