If every Web page were connected by hyperlinks to every other page, a crawler could index the entire Web. Once fetched by the crawler, a page is indexed. Most engines build an inverted index, in which each word points directly to all the documents it occurs in. Search engines also store information about word adjacency, words occurring in HTML titles and headers, and hyperlinks. AltaVista and some other search engines identify a page's language by comparing a profile of character strings in the document to one for documents in known languages.
Many Web pages have identical information, such as software documentation or news reports.
Deciding which copy is the original is harder. But in the case of wholesale site copying or theft, the older version is likely to be the real thing. When you query a search engine, the query terms, or keywords, are first compared to a list of pre-computed compound items. The more heavily weighted items will appear at the top of the search results. Google requires all query terms to be present on a page to be considered a match.