Their subject-oriented organization, while logical, makes it difficult to search for topics that aren't readily categorized. Also, this organization reflects only the primary subject of a page, ignoring its secondary and tertiary topics. Finally, indices catalog a smaller number of documents than most automated search engines, so results of a search in Yahoo, for example, are likely to be incomplete. Spiders, also known as robots or agents, are programs that scour the Web for resources and keep records of their findings. They begin at a single site, scan for links to other pages, and proceed onward, following a continuous stream of links across the Web.
Different spiders have different definitions of relevant, however.
Some venture into gopher or ftp space, gathering non-hypertext documents, while others restrict their wanderings to the Web proper. But the most powerful and useful engines allow you to search the entire text of a document. Any comprehensive search should include visits to at least two or three search engines. In many ways, Lycos dwarves the Web's first-generation search engines, but such older tools as WebCrawler , Nikos and the WWW Worm still hold their own, often suggesting links that don't turn up on a Lycos search. For one-stop spider shopping, you might want to bookmark the URL for CUSI , a unified search index for the Web.