Dark Matter on the Web Dark Matter on the Web Peter Bailey, Nick Craswell Dept. We introduce the concept of dark matter - information on the Web that is not or cannot be discovered by an individual or a search engine.
Dark matter seems an equally appropriate term for an analogous situation in Web space.
Its main contribution as a concept for the Web is in providing a language to describe how and why information is not discoverable. Our definition characterises why obtaining pages is dependent on the characteristics of the observer . Observers are either people who download pages from the Web using some browser technology, or crawler agents of certain kinds of Web search systems. Web crawlers find pages by following links from some seed set of URLs. But there are parts of the Web which are not visible from any of these search engines.