The Controller class takes care of URL filtering and scheduling, and interfaces with the Database to store the results of gatherer runs. The Reaper handles object fetching and summarising, getting workload from, and returning results to, the Controller. At various stages in the life cycle of an object, it may also include a content summary, some headers and assorted management information. Headers are information used within the system, and also those headers returned by the protocol that the URI is fetched by. Information contained in headers is transient, it is not stored persistently in any part of the system, and is not preserved across runs.
Management information is data used internally by the system to control the gathering process.
It is stored persistently when the object is saved, and is expected to be available when the object is retrieved from the database. All of this information is encapsulated by the Object class. There is purposefully no Harvest class to encapsulate the content summary, the intention being that the content summary in use is independent of the Harvest-NG code, and can be replaced. The Database class provides a wrapper around a number of options of database types. Currently both DBM and directory based databases are available.