Another related project underway is WebAnts aimed at creating cooperating explorers, so that an individual spider doesn't have to do all the work of finding things on the Web or duplicate other spiders' efforts. The The Harvest Information Discovery and Access System reaches beyond being merely a spider, but involves a series of subsystems to create an efficient, flexible, and scalable way to locate information. Harvest is an ambitious project to provide a way to create indexes and provide for efficient use of servers. Harvest is being designed and built by the Internet Research Task Force Research Group on Resource Discovery . It's intended to be a scalable form of infrastructure for building and distributing content, indexing information, as well as for accessing Web information.
Harvest consists of several subsystems.
A Gatherer collects indexing information and a Broker provides a flexible interface to this information. A user can access a variety of collections of documents. The Harvest WWW Broker , for example, includes content summaries of more than 7,000 Web pages. This databasa has a very flexible interface, providing search queries based on author, keyword, title, or URL-reference. Other subsystems further refine Harvest's capabilities.