The main emphasis of the program so far has been on the effective integration of advanced information retrieval and natural language processing capabilities since there are a number of important system integration issues and effectiveness and efficiency concerns that are critical to developing even a rudimentary pilot system. This distributed, modular approach provides flexibility in terms of the methods employed and in defining extensions to the baseline architecture. In a sense, MIDS is both an integration framework and a set of tools. Major Harvest components utilized include Gatherers and Brokers. A Broker provides a search interface to information collected by one or more Gatherers or other Brokers.
Harvest provides much of the system-framework for handling distributed information.
However, as discussed below, the basic Harvest system does not address well requirements for content routing to multiple Brokers where information cannot be a priori linked to a particular Broker, nor can it handle effectively query routing where broker level metadata is needed to effectively assign user queries to appropriate Brokers. These areas and others motivated a number of extensions to the basic Harvest system. It has an attribute-value format which is easily parsed and yet sufficiently expressive to handle many kinds of objects. Each Broker manages a set of topical areas and is periodically updated with SOIF records from the GDS based on the information type it desires. The BIS provides a topical browse service, enabling a user to retrieve documents based on topic selection.