Engines which are created automatically, such as HotBot, and Yahoo, which is created manually 1 . Automatically created engines create directory listings from bots and agents. Manually created engines use tools to formulate listings and then are fine tuned by humans. Spiders and Crawlers Search engines are comprised of three primary components. The first component is the spider or crawler, the second component is the index and the third component is the search engine software.
Indexing is the process of adding a web page to an index, similar to a directory, for later lookup.
Search engine software is the program or front end that accesses the index to find matches and later ranks those matches. How Search Engines Rank Web Pages Search engines rank web pages primarily by location and frequency 2 . Location is where the sought after word is situated on a page. Frequency is used to determine relevancy. Combinations that appear more frequently are deemed more relevant and are ranked higher.