The total number of documents read and retained in a database becomes much more important. For example, if you go to CNN's robot rules , you see that they disallow robot access to all sorts of things. This is primarily done to prevent spiders from skewing site traffic statistics. Some spiders are designed to pay a lot of attention to what the Web documents say in their titles. Other spiders are more concerned with words that appear near the top of the pages.
Search engines have their own specialties.
Com indexes articles from Usenet News. AltaVista allows you to translate documents that are in another languages. When I am looking for meteorological information from foreign countries, I find AltaVista very useful. I am not sure how practical this is, but it does minimize the usual pawing through hundreds of documents returned by most search engines. Some search engines appear to catalog documents like a directory service, and most directory services allow you to search though their categories with a search engine.