Internet Search Programs With so many documents on the web, a search mechanism was needed to catalog and retrieve the pages by subject. Your choice of search tool will differ based on the type of information you are looking for online. Some are reviewed and maintained by library or information professionals who will assign logical subject areas to web pages included in the database. Other may be computer generated indexes with only a key word ability that finds the words you type in all the lines of the documents in their database. Web robots are often used to search and index the web with very little human intervention.
Web Robots are programs that traverse the Web automatically.
Some people call them Web Wanderers, Crawlers, or Spiders. If an indexing robot knows about a document, it may decide to parse it, and insert it into its database. Some robots index the HTML Titles, or the first few paragraphs, or parse the entire HTML and index all words, with weightings depending on HTML constructs, etc. Some parse the META tag, or other special hidden tags. The directory's indexing system assigns specific subject terms to web pages describing the content.