This browsing mechanism helps clarify the retrieval results and allows users to access the value of information. The WWW is growing explosively, and today there are about 60,000,000 Web pages that Excite can gather from the Web. To efficiently examine this vast informational cyberspace, sophisticated new tools are being developed that automatically traverse the WWW, create a condensed summary of the contents that are found, and thus make the information readily available. Although this compressed information only approximates the full-text documents it represents, these tools nevertheless make it possible to manage and search through the vast amounts of information that is available on the Internet. The robot extracts hyperlinks from Web sites which it then uses to recursively travel to those URLs.
Information gathering is complete when all the links have been followed up.
Though this method may not be practical for collecting all the information on the Internet, it does allow the collection of a good proportion of the information that is available. The analysis consists of extracting those parts which are identified by HTML tags as being especially important. There are various kinds of documents on the Web. For example, there are indices such as Yahoo that contain an immense number of links to other Web sites, and there are pages that contain a great deal of information in and of themselves. The former measures the degree of unique informational content contained in a document and the latter determines to what extent a document is linked to other Web sites.