Some search engines include the opening portion of the text of Web pages in their lists, but others include only the titles or addresses of Web pages. WebCrawler uses a program called a spider to search the WWW for new documents and to index all the words in the documents. A person using WebCrawler enters a key word or phrase, and WebCrawler provides a list of all documents that contain the word or phrase. Each title is linked to the document's site on the WWW, so users can go directly from the list to any document on the list. A spider takes advantage of this structure by starting at one Web page and working its way out by following every link on a Web page and then following every link provided by the new Web pages.
Some spiders save the URL, or address, of every Web page they visit.
These spiders are used by search engines to build indexes of Web pages that users can access to search for information on a particular topic. Some spiders store only URLs of Web pages that have not been listed yet in order to update lists or provide lists of new Web pages. Some spiders make note of URLs that are no longer valid in order to correct lists.