Most sites have subject directories, e-mail capabilities, news, weather, stock updates, and lots of advertising. Use a subject directory if you have a broad topic or you want to browse a list of recommended sites. Use a search engine if you have a narrow or specific subject in mind or you want to search for particular types of documents. These programs are designed to go out into the World Wide Web and locate web pages, follow the links they offer, and take the information and add it to an index database. Webpage editors or professional webpage developers sometimes provide information about their pages as well.
Most spiders return to the sites on a regular basis to look for changes or updates.
Each search engine programs its spider differently so that no two spiders conduct a search the same. Therefore, the results you ll get will vary from engine to engine. For example, some spiders search for titles of web pages only, others look for particular words, or lines of text, still others reference every single word of a document. More on this topic Relevancy and ranking After you have entered your search terms, the request goes to the search engine index and an attempt is made to match your request with the information contained in the database. You receive back a list of sites that are generally listed in order of their relevancy to your search request.