Find out how you can mine these rich data sources. Huge portions of the Internet are invisible to most search engines. Some folks call this the invisible Web. Don't confuse the invisible Web with HTML pages that don't get indexed by the major engines. You can still find these pages by using specialty search engines .
Search engines use programs called spiders, which only index HTML pages.
Spiders can't index CGI scripts, PDF files, or information in databases. Various sources estimate you can find about 500 times more information in databases than in normal search engines. Instead of searching thousands of irrelevant categories on a search engine, you'll get more useful information by searching a single database where everything is related to your topic. This is like looking in a telephone book. If you're looking for Stephanie Jones in Chicago, it makes more sense to look only in the Chicago telephone book instead of looking through all telephone numbers worldwide.