The problem is that vast expanses of the Web are completely invisible to general purpose search engines like AltaVista, HotBot and Google. The simplicity of this format makes it easy for search engines to retrieve HTML documents, index every word on every page, and store them in huge databases that can be searched on demand. What's less easy is the task of actually finding all the pages on the Web.
Crawling is a resource-intensive operation.
It also puts a certain amount of demand on the host computers being crawled. For these reasons, search engines will often limit the number of pages they retrieve and index from any given Web site. It's tempting to think that these unretrieved pages are part of the Invisible Web, but they aren't. They are visible and indexable, but the search engines have made a conscious decision not to index them. In recent months, much has been made of these overlooked pages.