Lee Sorensen explained a search engine as an algorithm that can gather information and place it into a holding file for display. He described how search engines work, using a spider to go out and find the information and CGI to translate it for your display. It begins with a search form on your client machine. The engine then scans for the search term across a data file, placing the information into a result file on the client.
Engines harvest periodically.
Placing information into an index file speeds retrieval. Sorensen distinguished between the working of a local and an external search engine. Locally, indexing is done according to file location in a directory, rather than by file links, therefore it would not be affected by broken links as is possible with an external search. It does not harvest dynamic data such as CGI results. Sorensen said that designing pages with a generic title would result in a higher retrieval ranking.