Spiders and Crawlers Search engines are comprised of three primary components. The first component is the spider or crawler, the second component is the index and the third component is the search engine software. Indexing is the process of adding a web page to an index, similar to a directory, for later lookup. Search engine software is the program or front end that accesses the index to find matches and later ranks those matches. How Search Engines Rank Web Pages Search engines rank web pages primarily by location and frequency 2 .
Location is where the sought after word is situated on a page.
Frequency is used to determine relevancy. Combinations that appear more frequently are deemed more relevant and are ranked higher. Search engines produce different results because of the way they are tuned. Engines such as Exite throw in a popularity formula 2 . Web sites that are visited more often than others rank higher.