The third part of the model, the ontology, is used by the ingest agents to reduce the user interaction required to handle schema changes. Current warehouse solutions include monolithic centralized storehouses, or approaches based in multidatabase or federated database techniques. For a variety of reasons ranging from inflexibility to change, to heavy user burdens for accessing the underlying data, to performance that is heavily dependent on network and data source availability, none of these approaches are adequate for scientific data. DataFoundry is a combination of the above approaches. The schema is detailed enough to relieve the user burden, and contains information to help automate schema integration.
Data is kept both locally and in the external sources.
Decisions on which data to keep local is made based on maximum expected utility - or minimizing the query response times to common and important queries. Queries involving external data will be automatically translated into the external data site format before being dispatched. Data ingest will be based on intelligent agent technology with change detection techniques, resulting in a continuous and incremental update. DataFoundry is designed to be flexible enough to scale to the demands of realistic scientific environments, while providing a powerful query and inference capability across a wealth of data.