Each title is linked to the document's site on the WWW, so users can go directly from the list to any document on the list. A spider takes advantage of this structure by starting at one Web page and working its way out by following every link on a Web page and then following every link provided by the new Web pages. Some spiders save the URL, or address, of every Web page they visit. These spiders are used by search engines to build indexes of Web pages that users can access to search for information on a particular topic. Some spiders store only URLs of Web pages that have not been listed yet in order to update lists or provide lists of new Web pages.
Some spiders make note of URLs that are no longer valid in order to correct lists.