Find out how you can mine these rich data sources. How Search Engines Work Huge portions of the Internet are invisible to most search engines. Some folks call this the invisible Web. Don't confuse the invisible Web with HTML pages that don't get indexed by the major engines. You can still find these pages by using specialty search engines .
Search engines use programs called spiders, which only index HTML pages.
Spiders can't index CGI scripts, PDF files, or information in databases. Various sources estimate you can find about 500 times more information in databases than in normal search engines. You can find this invisible database information either through online directories or through a new all-encompassing search engine called ProFusion. Database directories These sites gather together online databases. You can't actually search the databases, but you can pinpoint just the databases you need.