This is very similar to the card catalog that a library uses. The main problem with an index is that it is not as extensive as other search engines because a new site is not indexed until a human indexer examines it. These are more or less successful depending on the word and the context of its use. These are computer programs that automatically search the web for new materials. These usually locate new resources more rapidly than an index, but the descriptions of the sites are not as accurate as an index since the program which catalogs them gathers its information only from the site, not from the skill of the indexer.
Some crawlers create their index by selecting their terms from the first few lines of text.
Others will index every word in a site. This can create an imprecise index that will generate many false responses. In concept based indexing the crawler indexes based on the context of a word in the sentence. This technique sounds good, but it is usually harder to implement than the search utilities will admit. However, there is no controlled vocabulary that the programmers use, so the resulting index is no more accurate than that generated by the crawler itself.