For a fee, InfoSeek will also let you search a bunch of handy non-Internet databases. It's all in the index Web spiders do more than just collect URLs. They also collect information about each page. The search engine's back-end software uses this information to create an index, which is what you're actually searching when you submit a query. Not surprisingly, indexing techniques vary from engine to engine.
Every engine indexes a page's URL and title.
Most engines also index the headers that start each section. Others record the most frequently mentioned words or the first few lines of text. Excite's concept-based indexing can find relevant pages even if they don't contain your specific keywords. While the size of the database determines the number of hits it delivers, the quality of the indexing is a major factor in determining how many of those hits are relevant to your search. WebCrawler returned 19 hits, compared with the more than 200 hits I got from InfoSeek.