Reaction from companies Kris Carpenter, director of search products and services for Excite, said her company purposely ignores a large part of the Web because of a lack of consumer interest. But he argued that users are seeking quality information, not merely quantity. If you want to find the best Thai food, and there are 14,000 results, the question isn't how many returns you got, but what are the top 10. The words on that new page are then indexed so the page can be found when a computer user begins a search. The crawling process helps the search engine compile an index made up of the most popular Web sites.
Crawling can unearth an enormous number of pages.
But storing, searching and delivering that amount of information would require a daunting volume of computer storage and high-speed connections to the Internet. How to be found Brown added that anyone who wants to be found can be, as most search engines allow people to submit their Web pages for manual inclusion in a search index. Commercial Web sites can also pay for prominent placement on some indexes. Excite's Carpenter said the future of search engines lies not in bigger indexes but in more specialized ones, in which having everything on a given subject could be indexed and displayed to viewers. Lumping everything into one big, be-everything index would be incredibly overwhelming.