The amount of material on the Web is enormous, larger than most databases. This leads to a variety of operational scalability problems. Unlike a structured database for which some number of people are authorized to enter and update data, Web pages require discovery and periodic revisits. Rate of change. Web pages change at many different rates, from daily to never.
New pages come online at a prodigous rate .
While page authors may request an updated visit, discovery of changes and new pages is up to the search engine. Infoseek lets the search engine sysadmin set the maximum and minimum revisit intervals and Infoseek then places pages into one of six queues according to the last modified date on the Web page. Consider the UNH Intranet, with more than 32000 pages known to Infoseek. Search engines build an index of all the material discovered. The search engine index, lacking a pre-defined structure, uses other techniques to aid in a search.