With our growth rates, we need that. Sun enables rapid growth and the delivery of new services. Infoseek has databases totaling 170 GB on a Sun Storage TM Array. Kirsch says, Weve had to scale up our systems very rapidly as our user traffic increases. Sun and the Sun servers work very well for that.
Spiders index the Web.
Infoseeks Internet navigation services search pre-generated indices stored in Informix databases. The company creates these indices using automated software agents nicknamed spiders that crawl across the Web from URL to URL and acquire descriptive information. There are many ways to do this, but Infoseeks spider uses the most demanding approach, recording the full text of pages from the sites visited. This text is organized into multiple indices, which are then updated periodically with the results of the latest Web crawl. When a user initiates a request, the search engine looks through the indices for matches.