With a query filter, you can perform your keyword search against a very detailed subset of the entire dejanews database. This can be an invaluable tool to get a sense of what that person is all about. Organizations can also use Dejanews to see if their employees have been conforming to the organization's Access policy with regards to the Newsgroups. You can even tell which search engines Yahoo keeps out in their robots.txt file. See The Web Robot Pages and Robot Exclusion Standard Revisited for details.
Some sites require a password .
If you have to be a registered user to access a site - then that site's content will be missed by the search engine. For example, the vast majority of the Wall Street Journal is not searchable through the search engines. Many web pages are generated by database queries . For example, at Amazon Books most of the information about 1 million books exists in their database. The web pages describing a particular book, author, publisher, etc are only created in response to User queries.