All of the above. The problem is that the Web is just getting too vast it now consists of well more than a billion pages and is growing, according to the NEC Research Institute s C. Lee Giles, author of studies on the accessibility and distribution of information on the Web. There must be at least five times as much as what s indexed by the search engines. Now a portal with all kinds of bells and whistles, at Yahoo s heart is a human-made categorized directory of more than a million sites on all subjects.
Indexes grow But humans just couldn t keep up on their own.
Yahoo s concept of one big, categorized list of bookmarks needed to be backed up with searchable, computer-generated indexes of all the words on all the pages collected by spider programs crawling the Web, such as AltaVista, Go, Northern Light and the Inktomi index fueling HotBot, MSN and other search services. But as these computer-driven indexes grew in size to cover claims of up to 500 million pages, the results of simple word searches became unwieldy and frustrating for even savvy computer users. So the search scientists brought the people s choices back into the programs. Google, the favorite computer-made search engine, according to an informal survey of more than 40 professional news researchers, uses the Web community s expertise in its page ranking. Just as you trust the links on a really good site to get you to other good pages, Google crawls the Web scooping up hyperlinks and uses them to figure out how important a page is by what s pointing to it.